EBWiki Backup S3 production bucket periodically

We need to figure out a good way to backup our uploaded images bucket.

Here is a good discussion of the alternative approaches: http://stackoverflow.com/questions/17832860/backup-strategies-for-aws-s3-bucket

copying to another s3 bucket would seem to meet our needs for protection. It does make sense to think of what exactly we are protecting ourselves from. eg. backing up to another s3 bucket will not protect us against a general failure of AWS or S3.

Jul 12 '16 14:07 campgurus

I set up a backup bucket called blackops-backup, but have not automated the process yet.

Jul 14 '16 19:07 campgurus

What is the manual version of the backup process and can we place that on the wiki (for the wiki; ha!)

Jul 15 '16 10:07 trystant

Added to the wiki

Jul 16 '16 03:07 campgurus

@campgurus @trystant So are we going into the production console and running this command? Writing a rake task or background job that does this would be good for Summer of Code people.

Mar 02 '18 07:03 rlgreen91

we're not...this should be a good SoC project.

May 20 '18 08:05 trystant

Regarding the backup, s3 is very durable. If you have versioning turned on, it is even better. Do you have another cloud that you would like to hold these in? Backing up to a second s3 bucket with aws s3 sync would help against accidental or intentional deletions. If the fear is complete aws meltdown, then you would need to run the sync to a local vm on another cloud provider to keep a copy of the data.

Oct 10 '19 23:10 gt50

I agree that S3 is very durable and it would still be helpful to:

create or use a program that would sync to a backup s3 bucket. Could be a rake task
create or use a program that would sync to another cloud backup solution (digital ocean, cloudinary, etc)
create or use a program to schedule the runs of these other tasks.

Care to evaluate our S3 bucket @gt50?

Oct 11 '19 12:10 trystant

No problem @trystant. Set up a u/p with access to the bucket in IAM, and I can take a look.

Depending upon the size of the bucket there are many ways to deal with this. A very small instance on another cloud could run an "aws s3 sync" command to keep a local copy of all of the data. This is probably the simplest option. Another option is using lambda to places copies of any file that is place in the bucket in other locations.

I just checked out DO's offerings. I think Spaces would be perfect for this.

Oct 11 '19 21:10 gt50