Backing up Object Store

Object Store data, while stored redundantly via Ceph, is not backed up by default. Object Store buckets are backed up to the TSM tape system upon request only by following the procedures listed on this page.

Content and policies

The following are the default contents and policies of requested backups:

What is in the backup?

* Only the bucket data is backed up.  We are not currently backing up the IAM policies of the buckets or objects.
* Only the current version of the data is seen by the backup system.  Object chunks or versioned objects are not seen by the backups system.

What is the backup policy?

* Backups are run on a daily basis.
* The current object and one modified version of object are kept (this is different than full bucket versioning).    
* The modified version is kept for 6 month - after that period only the current object is kept. 
* Deleted objects are kept for 6 months.

Backup Procedure

Please follow this procedure to request backups of your buckets.

Email the list of buckets

Send a list of buckets to be backed up to sd4h support with the name and ID of the project where the buckets live.

Give us permission

An IAM policy statement must be applied to all the buckets you want to backup so the TSM robot user in charge of the backup can access them. This can be done with the aws cli.

First, ensure that my-bucket currently has no IAM policy. Check bucket my-bucket using profile my-profile (as defined in ~/.aws/config and ~/.aws/credentials files) with:

$aws s3api get-bucket-policy --profile c3g-data-repos --bucket my-bucket

An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist

If that command returns something, the new policy statements must be added to the existing policy (which is not covered here).

The following policy.json needs to be applied.

File : policy.json

{
"Statement": [
  {
    "Effect": "Allow",
    "Principal": {"AWS": ["arn:aws:iam:::user/tsm"]},
    "Action": [
      "s3:ListBucket",
      "s3:GetObject"
    ],
    "Resource": [
      "arn:aws:s3:::my-bucket/*",
      "arn:aws:s3:::my-bucket"
    ]
  }
]
}

Load the policy onto the bucket my-bucket using the profile my-profile:

$aws s3api put-bucket-policy --policy file://my-policy.json --profile my-profile --bucket my-bucket

Confirm IAM policy applied

As we did before, request the bucket's IAM policy, ensuring that the contents of policy.json are listed.

$aws s3api get-bucket-policy --profile c3g-data-repos --bucket my-bucket

Restore Procedure

List bucket

Send us list of buckets or object to restore to sd4h support.

Give us permission

For each bucket you want to be restored, you will be asked to create a bucket with the -restore suffix.

Following the previous procedure, apply this restore IAM policy.

File : policy.json

{
"Statement": [
  {
    "Effect": "Allow",
    "Principal": {"AWS": ["arn:aws:iam:::user/tsm"]},
    "Action": [
      "s3:ListBucket",
      "s3:GetObject",
      "s3:PutObject",
      "s3:PutObjectAcl",
      "s3:AbortMultipartUpload"
    ],
    "Resource": [
      "arn:aws:s3:::my-bucket-restore/*",
      "arn:aws:s3:::my-bucket-restore"
    ]
  }
]
}

Once done, we will restore your data to the *-restore buckets.