Backup to Amazon S3 with Duplicity

  • Duplicity creates tar archives encrypted with GPG.
  • Duplicity uses rsync to create the tar archives.
  • Duplicity’s power comes from it’s ability to create backups stored on Amazon S3 Buckets.

Install Duplicity

On Ubuntu

apt-get -y install python-boto
apt-get -y install duplicity
apt-get -y install gnupg

CentOS

  • Duplicity is available from Available from the [http://fedoraproject.org/wiki/EPEL EPEL Repository]
  • Python-Boto is available from the [http://wiki.centos.org/AdditionalResources/Repositories CentOS-Extras Repository]
  • GPG (GnuPG) is available from the [http://wiki.centos.org/AdditionalResources/Repositories CentOS-Extras Repository]
yum -y install python-boto
yum -y install duplicity
yum -y install gnupg2

Setup GPG if you have not already

  • You can generate a new key for your backups, or import existing GPG keys.

Generate a GPG Key

gpg --gen-key

Export an existing key

  • List the Keys
gpg --list-keys
  • The key will be listed according to the bits and the 8-bit hex code
pub   4096R/3977EFB8 2010-12-22
uid                  Devon Hubner <devon@hubner.org>
sub   4096R/C19C8706 2010-12-22
  • Specify the 8-bit hex id of the key you want to export: 3977EFB8
gpg --armor --output pubkey.txt --export 3977EFB8
gpg --armor --output secretkey.txt --export-secret-key 3977EFB8

Import an existing key to use for the backups

gpg --import pubkey.txt 
gpg --import secretkey.txt
  • You need to validate the key to use it.
  • Edit the key and use the trust command.
gpg --edit devon@hubner.org
Command> trust

Secret key is available.

Command> trust

Please decide how far you trust this user to correctly verify other users keys
(by looking at passports, checking fingerprints from different sources, etc.)

  1 = I dont know or wont say
  2 = I do NOT trust
  3 = I trust marginally
  4 = I trust fully
  5 = I trust ultimately
  m = back to the main menu

Your decision? 5
Do you really want to set this key to ultimate trust? (y/N) y

Command> quit

Duplicity Backup Commands

full: full backup even if previous signatures exist

duplicity full ${DEST} ${SRC}

incremental: files changed since the previous signature

duplicity incremental ${DEST} ${SRC}

verify: log a message for every file that has changed

duplicity verify ${DEST} ${SRC}

restore: restore files

duplicity restore ${DEST} ${SRC}

Backup to S3 Test

Create a directory and file(s) to backup

mkdir backmeup
echo "hello" > backmeup/hello.txt

FILE: duplicity_test.sh

#!/bin/bash

export AWS_ACCESS_KEY_ID=******************************
export AWS_SECRET_ACCESS_KEY=**************************
export PASSPHRASE=************************

GPG_KEY=3977EFB8
# gpg --list-keys
# pub   4096R/3977EFB8 2010-12-22
# uid                  Devon Hubner <devon@hubner.org>
# uid                  [jpeg image of size 2993]
# sub   4096R/C19C8706 2010-12-22
# sub   4096g/9CCDB062 2010-12-22

# The source of your backup
SOURCE=backmeup/

# The destination
# Note that the bucket need not exist
# but does need to be unique amongst all
# Amazon S3 users. So, choose wisely.
S3_BUCKET=dhubner_bucket1
DEST=s3+http://${S3_BUCKET}

##### Purge backups older than 1 Month ###

# TIME FORMATS
# s: seconds
# m: minutes
# h: hours
# D: days
# W: weeks
# M: months
# Y: years

duplicity remove-older-than 1M \
  --encrypt-key=${GPG_KEY} \
  --sign-key=${GPG_KEY} \
  --force \
  s3+http://ss1backup

#########################################

duplicity full \
 --encrypt-key=${GPG_KEY} \
 --sign-key=${GPG_KEY} \
 ${SOURCE} ${DEST}

#########################################

Run the Duplicity Backup Test

./duplicity_test.sh
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
--------------[ Backup Statistics ]--------------
StartTime 1324436026.35 (Tue Dec 20 21:53:46 2011)
EndTime 1324436026.37 (Tue Dec 20 21:53:46 2011)
ElapsedTime 0.02 (0.02 seconds)
SourceFiles 2
SourceFileSize 4102 (4.01 KB)
NewFiles 2
NewFileSize 4102 (4.01 KB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 2
RawDeltaSize 6 (6 bytes)
TotalDestinationSizeChange 1872 (1.83 KB)
Errors 0
-------------------------------------------------

Look for the Duplicity Backups on Amazon S3

Troubleshooting

Duplicity Error: There is no assurance this key belongs to the named user

Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
No old backup sets found, nothing deleted.
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
Last full backup is too old, forcing full backup
GPGError: GPG Failed, see log below:
===== Begin GnuPG log 
gpg: 5693592F: There is no assurance this key belongs to the named user
gpg: [stdin]: sign+encrypt failed: unusable public key
===== End GnuPG log 
  • Solved by validating the key with command: gpg –edit 5693592F
categories: aws |