Tagged as:

It’s dangerous to admin alone! Take this etckeeper.

It’s the end of a long day’s coding, you’re sipping a hard-earned frosty beer, when SUDDENLY… your phone starts blowing up due to a new (!) and never-before-seen (!!) server issue. “What?? How did that change?,” you think, stress mounting as you pull out your laptop and SSH in. You remembered changing some config somewhere in /etc… but what changed? How quickly can you restore your system to a sane state?

How would you like to have this?

you@problem-host ~> cd /etc
you@problem-host ~> sudo git diff HEAD~

Rather than spend hours digging through logs and pouring your problems into Google, how’d you like to have a line-by-line diff of the most recent config changes?

Oh hey, that’s right, you changed the Apache configuration. Let’s revert that change and see if it fixes it.

you@problem-host /etc> sudo git checkout httpd/httpd.conf
you@problem-host /etc> sudo apachectl graceful

Ah, that was it! Your site is back up, your beer still frosty — and your evening devoid of server hell. Tackling the rest of the problem can wait until tomorrow.

You want in on this? Lemme tell you about my homeboy etckeeper.

meet etckeeper

Etckeeper takes the VCS of your choice and throws all of your /etc/ files into it. It also hooks into your distro’s package management. BEFORE you install a package, Etckeeper checks in your latest config changes. AFTER yum or apt finishes their business, etckeeper checks in the new configurations added by the package.

The result?

abcd1234  committing changes before apt run
cdab5412  committing changes after installing openssl-0.9.8k

Pretty neat, eh!

install etckeeper & VCS of your choice

Now, I prefer git, because it’s what I use for my everyday development work. That said, etckeeper is flexible and allows you use bzr, hg, darcs, or git.

The Ubuntu package defaults to Bazaar. Instruct it to use git by installing like so:

you@host > sudo aptitude install etckeeper git-core

(I’m not a RedHat guy, but this tutorial looks pretty great.)

Lots of messages will fly by, including the initial commit. You’re done!

Try running git log in /etc to see what just happened:

you@host ~> cd /etc
you@host /etc> sudo git log --summary
commit fb5886961239edc706567b564578228cfe3dd9db
Author: deploy <deploy@li179-98.members.linode.com>
Date:   Wed Apr 24 07:58:58 2013 -0400

Initial commit

create mode 100755 .etckeeper
create mode 100644 .gitignore
create mode 100644 adduser.conf
create mode 100644 aliases
create mode 120000 alternatives/Mail
create mode 120000 alternatives/Mail.1.gz
create mode 100644 alternatives/README
create mode 120000 alternatives/aptitude
create mode 120000 alternatives/awk
create mode 120000 alternatives/awk.1.gz


some new tricks

Check out what you just made possible — previously mysterious and difficult questions are now a piece of cake.

What did /etc/password look like last week?

you@host /etc> sudo git diff '@{last week}'

What the heck did that package just change?

you@host /etc> sudo git diff HEAD~

Crap, I screwed up interfaces

Let’s revert that to a couple of commits ago:

you@host /etc> sudo git checkout HEAD~~ network/interfaces ...

Suddenly, all of the time you put into learning git starts paying off in your systems work. System configuration files? Just like writing and managing code. You got this.

The best part is that etckeeper is a snap to get going, won’t even take you two minutes.


If you’re like most organizations, your servers are somewhere between “hand configured” and “totally awesome Cadillac automation.” etckeeper works great all along the spectrum.

In your home-spun hand tailored server configs, you get incredibly detailed changes and the ability roll back. When you’re doing full-on configuration management, you can easily test & undo the inevitable messed up recipe.

In any case you get to use tools you know (i.e. git) to make your workflow better, cleaner, and — dare I say — more fun.

All of that for two minutes’ work!

Hey — thanks for reading!

My name is Fred, and I'm a web developer by trade, Linux sysadmin by necessity. I want you to win at hosting your own web applications.

Server administration doesn't have to come with a side of stomach ulcer.

As a developer you've got most of the skills you need, all you need are some practical ways to up your server game.

Questions? Email me.