Backing up AWS S3 bucket to a local machine

There are lots of solutions that talk about making a copy of a local folder and sending it to an S3 bucket. The solution I’ve needed is the reverse of this; creating a local backup of an bucket.

TlDr;

Download s3tools onto your server and configure it… Details on configuring are below.

1. Create an AWS IAM User

In the Security Credentials of the AWS console create a new IAM user. The user policy only needs read-only access to S3.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "s3:Get*",
        "s3:List*"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

Make sure you note down the access keys, we’ll need them later.

2. Download Amazon S3 Tools

Amazon S3 Tools is a simple command line script that allows read / write access to S3 buckets.

You can download it via their website, or via your flavour of Linux repository. For Ubuntu that is just sudo apt-get install s3cmd

3. Configure

Once installed you need to tell S3 Tools the access keys to Amazon, and a couple of other configuration settings. This is started by running s3cmd —configure. You will be asked for:

• the access key,
• secret,
• a password (to encrypt the local config, make one up and note it down),
• a path for GPG (just press return for default),
• use HTTPS (a good idea),

Once the settings are entered you can then test the connection, and if everything works the settings can be saved.

4. Download your bucket

S3 Tools has a lot of options to go back and forth between S3. A couple to note:

s3cmd ls - Lists your buckets
s3cmd get s3://[bucket-name]/* - Download your bucket

In my case I want to pull a directory within a bucket down, and if the files have changed over write them. The command I am using is:

s3cmd sync --check-md5 --force -r s3://[bucket-name]/[directory-name]/ /home/[backup-location]/

A quick explanation:

• sync - Synchronises the directory
• check-md5 - look at the md5 hash of a file to see if it has changed

• force - force overwrite the changed files rather than asking for permission

  -r - do it recursively

It is worth noting that if you want to do a test run before writing data you can add in the dry-run option which will output what it would be doing, but not actually run it.

5. Optional extra…

To create a historical record you can then use a solution like Bacula to keep various versions of the folder. However, that is the topic of another blog.