Ec2 Backups

From Notes

Backing up an EC2 instance can be quite simple... it can also be quite difficult. Just depends as to what you want to backup and where you want to store those backups. This article details a few of the very simple methods I've pieced together to ensure I have stable and consistent backups of my EC2 instance.

Contents 1 ext2/3/4 dump 2 MySQL Dump 3 AMI Bundle 4 s3-bash

ext2/3/4 dump

This script dumps the root file system to a bzip'ed file and then uploads that file to s3 using a super nifty script s3-bash. This code can use A LOT of improvement: error checking, sanity checks, bail outs, clean up, log rotation, etc...

## File: dump-vol1.bash 
#!/bin/bash

DATE=`date +"%Y%m%d"`
BACKUP=/backups/vol1.ext4.dump.$DATE.bz2
LOG=/var/log/vol1-backup-$DATE.log

(
echo "Begin backup /vol1: "`date`
echo Backing up vol1 to $BACKUP
/sbin/dump -0 -f -  /mnt/vol1| /usr/bin/bzip2 > $BACKUP
echo Finished: `date`
echo
ls -l $BACKUP
/usr/local/s3bash/putFile -f $BACKUP -b bucket
if [ $? -eq 0 ]; then
	rm -f $BACKUP
fi
) > $LOG
mail -s "vol1 backup `date`" my.email@addr < $LOG && rm -f $LOG

MySQL Dump

Backing up the MySQL database is very important! Same method (and problems) as the above ext dump. It should be noted that while this script does order a freeze of the MySQL dbases, better/other methods should be considered for higher utilization systems.

## File: mysqlbackup.bash 
#!/bin/bash

DATE=`date +"%Y%m%d"`
BACKUP=/backups/MySQLDump-$DATE.mysql.bz2
LOG=/var/log/mysqlbackup-$DATE.log
(
echo "Beginning MySQLDump: "`date`
echo Backing up mysql databases to $BACKUP
/usr/bin/mysqldump --all-databases --add-drop-database -F -l | bzip2 > $BACKUP
echo Finished: `date`
echo
ls -l $BACKUP
/usr/local/s3bash/putFile -f $BACKUP -b bucket
if [ $? -eq 0 ]; then
	rm -f $BACKUP
fi
) > $LOG
mail -s "mysqldump `date`" my.email@addr < $LOG && rm -f $LOG

AMI Bundle

One of the awesome things about AWS EC2 instances is the AMI deployment system. Using the below script, you can build a custom AMI of your running system. In the even of a total catastrophe, you can quickly and easily bring up a new EC2 instance using your AMI as the base.

# ec2-bundle-vol  \
 -c /opt/aws/creds/AWS-x590-Certificate-2011.03.23.pem \
 -k /opt/aws/creds/AWS-x590-PrivateKey-2011.03.23.pem \
 -u xxxx-xxxx-xxxx \
 -d /tmp/bundle \
 --exclude /mnt

Uploads the above created AMI to S3

# ec2-upload-bundle \
 -b bundle-backups \
 -m /tmp/bundle/image.manifest.xml \
 -a `cat /opt/aws/creds/AWS-access-key-id` \
 -s `cat /opt/aws/creds/AWS-secret-key`

s3-bash

s3-bash is a super nifty collection of tools which allows you to upload files, using 100% bash, to s3. This is pretty damn cool; noting that the REST API requires HMAC-SHA1 signed auth keys to be included in the cURL header. To make uploads of my backups painless, I've created a wrapper around the s3-put command. As mentioned above, this script requires some sanity and error checking; but otherwise, it seems to function well.

## File: /usr/local/s3bash/putFile
#!/bin/bash
AWSKey="myAWSKey"
AWSSecretKeyFile="/opt/aws/creds/AWS-secret-key"
access="private"
s3put="/usr/local/s3bash/s3-put"


function printHelpAndExit {
	echo "$0 params:"
	echo "-h  print this help message"
	echo "-f [file to upload]"
	echo "-b [S3 bucket to upload to]"
	exit $1
}

while getopts "hf:b:" optionName; do
	case "$optionName" in
		h)	printHelpAndExit 0;;
		f)	fqpn="$OPTARG";;
		b)	bucket="$OPTARG";;
	esac
done

if [ ! -f $fqpn ]; then
	echo "$fqpn not found"
	printHelpAndExit 0
fi

if [ -z $fqpn ]; then
	echo "you must specify a file to upload (-f)"
	printHelpAndExit 1
fi

if [ -z $bucket ]; then
	echo "you must specify a bucket (-b)"
	printHelpAndExit 1
fi

filename="$(basename $fqpn)"
contentType="$(file --mime-type $fqpn | awk '{print $2}')"

$s3put \
  -k $AWSKey \
  -s $AWSSecretKeyFile \
  -S \
  -T $fqpn \
  -c "$contentType" \
  /$bucket/$filename?$access
exit $?