169
edits
Backing up Object Store: Difference between revisions
→Give us permission
m (Confirm policy applied section added.) |
|||
| (6 intermediate revisions by 3 users not shown) | |||
| Line 1: | Line 1: | ||
__FORCETOC__ | __FORCETOC__ | ||
Object Store data, while stored redundantly via Ceph, is not backed up. Object Store buckets are backed up to the TSM tape system upon request only by following the procedures listed on this page. | Object Store data, while stored redundantly via Ceph, is not backed up by default. Object Store buckets are backed up to the TSM tape system upon request only by following the procedures listed on this page. | ||
The following contents and policies | == Content and policies == | ||
The following are the default contents and policies of requested backups: | |||
What is in the backup? | What is in the backup? | ||
| Line 11: | Line 12: | ||
* Backups are run on a daily basis. | * Backups are run on a daily basis. | ||
* The current object and one modified version of object are kept (this is different than full bucket versioning). | * The current object and one modified version of object are kept (this is different than full bucket versioning). | ||
* The modified version is kept for | * The modified version is kept for 60 days - after that period only the current object is kept. | ||
* Deleted objects are kept for 6 months. | * Deleted objects are kept for 6 months. | ||
| Line 20: | Line 21: | ||
==Email the list of buckets== | ==Email the list of buckets== | ||
Send a list of buckets to | Send a list of buckets to be backed up to [mailto:juno@calculquebec.ca sd4h support] with the name and ID of the project where the buckets live. | ||
==Give us permission== | ==Give us permission== | ||
An | An IAM policy statement must be applied to '''all the buckets''' you want to backup so the TSM robot user in charge of the backup can access them. This can be done with the [https://awscli.amazonaws.com/v2/documentation/api/latest/index.htm aws cli]. | ||
First, ensure that <code>my-bucket</code> currently has no policy. Check bucket <code>my-bucket</code> using profile <code>my-profile</code> (as defined in ~/.aws/config and ~/.aws/credentials files): | First, ensure that <code>my-bucket</code> currently has no IAM policy. Check bucket <code><my-bucket></code> using profile <code>my-profile</code> (as defined in ~/.aws/config and ~/.aws/credentials files) with: | ||
<pre>$aws s3api get-bucket-policy --profile c3g-data-repos --bucket my-bucket | <pre>$aws s3api get-bucket-policy --profile c3g-data-repos --bucket <my-bucket> | ||
An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist | An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist | ||
| Line 45: | Line 46: | ||
"Action": [ | "Action": [ | ||
"s3:ListBucket", | "s3:ListBucket", | ||
"s3:GetBucketPolicy", | |||
"s3:GetObject" | "s3:GetObject" | ||
], | ], | ||
"Resource": [ | "Resource": [ | ||
"arn:aws:s3:::my-bucket/*", | "arn:aws:s3:::<my-bucket>/*", | ||
"arn:aws:s3:::my-bucket" | "arn:aws:s3:::<my-bucket>" | ||
] | ] | ||
} | } | ||
| Line 56: | Line 58: | ||
</syntaxhighlight> | </syntaxhighlight> | ||
Load the policy onto the bucket <code>my-bucket</code> using the profile <code>my-profile</code> | Load the policy onto the bucket <code><my-bucket></code> using the profile <code>my-profile:</code> | ||
<pre> | <pre> | ||
$aws s3api put-bucket-policy --policy file://my-policy.json --profile my-profile --bucket my-bucket | $aws s3api put-bucket-policy --policy file://my-policy.json --profile my-profile --bucket <my-bucket> | ||
</pre> | </pre> | ||
== Confirm policy applied == | == Confirm IAM policy applied == | ||
As we did before, request the bucket's IAM policy, ensuring that the contents of policy.json are listed. | As we did before, request the bucket's IAM policy, ensuring that the contents of policy.json are listed. | ||
$aws s3api get-bucket-policy --profile c3g-data-repos --bucket my-bucket | $aws s3api get-bucket-policy --profile c3g-data-repos --bucket my-bucket | ||
| Line 70: | Line 72: | ||
==List bucket== | ==List bucket== | ||
Send us list of buckets or | Send us the list of buckets or objects to restore at [mailto:juno@calculquebec.ca sd4h support]. | ||
==Give us permission== | ==Give us permission== | ||
For each bucket you want to be restored, you will be asked to create a bucket with the <code>-restore</code> suffix. | |||
Following the previous procedure, apply this restore IAM policy. | |||
<div class="filename">'''File :''' policy.json </div> | <div class="filename">'''File :''' policy.json </div> | ||
<syntaxhighlight lang="json" file="my-policy.json"> | <syntaxhighlight lang="json" file="my-policy.json"> | ||
| Line 91: | Line 95: | ||
"Resource": [ | "Resource": [ | ||
"arn:aws:s3:::my-bucket-restore/*", | "arn:aws:s3:::my-bucket-restore/*", | ||
"arn:aws:s3:::my- | "arn:aws:s3:::my-bucket-restore" | ||
] | ] | ||
} | } | ||
| Line 98: | Line 102: | ||
</syntaxhighlight> | </syntaxhighlight> | ||
Once | Once done, we will restore your data to the <code>*-restore</code> buckets. | ||