Backing up AWS S3 bucket to a local machine

05 February 2015

There are lots of solutions that talk about making a copy of a local folder and sending it to an S3 bucket. The solution I’ve needed is the reverse of this; creating a local backup of an bucket.

TlDr;

Download s3tools onto your server and configure it… Details on configuring are below.

1. Create an AWS IAM User

In the Security Credentials of the AWS console create a new IAM user. The user policy only needs read-only access to S3.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "s3:Get*",
        "s3:List*"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

Make sure you note down the access keys, we’ll need them later.

2. Download Amazon S3 Tools

Amazon S3 Tools is a simple command line script that allows read / write access to S3 buckets.

You can download it via their website, or via your flavour of Linux repository. For Ubuntu that is just sudo apt-get install s3cmd

3. Configure

Once installed you need to tell S3 Tools the access keys to Amazon, and a couple of other configuration settings. This is started by running s3cmd —configure. You will be asked for:

Once the settings are entered you can then test the connection, and if everything works the settings can be saved.

4. Download your bucket

S3 Tools has a lot of options to go back and forth between S3. A couple to note:

s3cmd ls - Lists your buckets
s3cmd get s3://[bucket-name]/* - Download your bucket

In my case I want to pull a directory within a bucket down, and if the files have changed over write them. The command I am using is:

s3cmd sync --check-md5 --force -r s3://[bucket-name]/[directory-name]/ /home/[backup-location]/

A quick explanation:

It is worth noting that if you want to do a test run before writing data you can add in the dry-run option which will output what it would be doing, but not actually run it.

5. Optional extra…

To create a historical record you can then use a solution like Bacula to keep various versions of the folder. However, that is the topic of another blog.

Back to all posts