OfflineIMAP claims to be faster, but I haven’t really checked it (and I’m not sure how important that is giving that it runs in the background). From what I saw configuring them is mainly a task of cut-and-paste, but getmail requires to list every label you want to backup, which I consider is a major downside. As both are able to save the mails to maildir format, it should be easy to back it up using duplicity.

Conclusion: This was a short comparison, mainly to guide me in choosing the right backup for me, you may have different opinions (which, of course, I would gladly hear). I finally chose OfflineIMAP, mainly due to the labels issue.

Note on desktop clients: It seems that every decent one can be configured to work with a local maildir, so you can use them to read the backups. As I prefer Gmail’s interface, I will only use desktop clients in case I’m offline, so read-only access from desktop client seems good enough for me.

Will start by creating a simple wrapper to duplicity:

#! /usr/bin/python
import sys
import os
 
duplicity_bin = '/usr/bin/duplicity'
 
env = {
    'AWS_ACCESS_KEY_ID':     'PUT YOUR KEY ID HERE',
    'AWS_SECRET_ACCESS_KEY': 'PUT YOUR SECRET ACCESS KEY HERE',
    'PASSPHRASE':            'PUT ENCRYPTION PASSPHRASE',
}
env.update(os.environ)
 
os.execve(duplicity_bin, sys.argv, env)

save this under dupicity-wrapper.py and chmod 0500 it so only you will be able to read and execute it.

Note: You’ll want to write down the passphrase and store it in a safe location (preferably in two separate locations). So in case you’ll need to restore the backups, you won’t have useless encrypted files.

Now edit your crontab and add a line like the following:

10 1 * * 0 /path/to/duplicity-wrapper.py /path/to/folder/ s3+http://bucket-name/somefolder &>> ~/log/backups.log

This will create a weekly backup for /path/to/folder. The backup will be encrypted with what ever passphrase you’ve given in the duplicity-wrapper.py. The output of the backup process will be saved into ~/log/backups.log.

You should also run

/path/to/duplicity-wrapper.py full /path/to/folder/ s3+http://bucket-name/somefolder

in order to create full backups. You might want to periodically verify your backups:

/path/to/duplicity-wrapper.py collection-status s3+http://bucket-name/somefolder
/path/to/duplicity-wrapper.py verify s3+http://bucket-name/somefolder /path/to/folder/

To check the status of the backups and to verify them.

And last but not least, in case you ever need the backups, you can restore them using:

/path/to/duplicity-wrapper.py restore s3+http://bucket-name/somefolder /path/to/folder/

Security Considerations

As I know, some people will comment on the saving of the encryption passphrase plainly in a file, I will explain my reasoning. I use the above encryption in order to secure my files in case of data leakage from Amazon S3. In order to read my backups, or silently temper with them, some on will have to get the passphrase from my machine. While, this isn’t impossible, I will say it’s unlikely. Furthermore, if someone has access allowing him to read files from my computer, he doesn’t need the backups, he can access the files directly.

I’ve given some thought about making the backups more secure, but it seems you always have to compromise on either automation or incremental backups. But, as I wrote, the current solution seems to me strong enough given the circumstances. Nonetheless, if you’ve got a better solution it would be nice to hear.

The first one is that one must export the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY or else you get a error message from the underlying boto library:

  File "//usr/lib64/python2.5/site-packages/boto/connection.py", line 148, in __init__
    self.hmac = hmac.new(self.aws_secret_access_key, digestmod=sha)
AttributeError: S3Connection instance has no attribute 'aws_secret_access_key'

The second thing to note is that is the way to specify buckets for duplicity. Instead of

s3://<bucket-name>/<prefix>

which is used by s3cmd (which is a great tool), it asks to specify it using

s3+http://<bucket-name>/<prefix>

Being aware of these points can save some time and frustration. In order to automate the backup process one can use cron. For example add the following to your crontab

AWS_ACCESS_KEY_ID="<your-key-id>"
AWS_SECRET_ACCESS_KEY="<your-secret-access-key>"
0 1 * * 0  duplicity --no-encryption <folder-to-backup> s3+http://<bucket-name>/<prefix> &>> ~/backups.log
AWS_ACCESS_KEY_ID=""
AWS_SECRET_ACCESS_KEY=""

I unset the environment variables, so it won’t leak to other cron jobs unnecessarily.