5
edits
Backing up Object Store: Difference between revisions
no edit summary
No edit summary |
|||
| (2 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
__FORCETOC__ | __FORCETOC__ | ||
Object Store data, while stored redundantly via Ceph, is not backed up. Object Store buckets are backed up to the TSM tape system upon request only by following the procedures listed on this page. | Object Store data, while stored redundantly via Ceph, is not backed up by default. Object Store buckets are backed up to the TSM tape system upon request only by following the procedures listed on this page. | ||
The following contents and policies | == Content and policies == | ||
The following are the default contents and policies of requested backups: | |||
What is in the backup? | What is in the backup? | ||
| Line 11: | Line 12: | ||
* Backups are run on a daily basis. | * Backups are run on a daily basis. | ||
* The current object and one modified version of object are kept (this is different than full bucket versioning). | * The current object and one modified version of object are kept (this is different than full bucket versioning). | ||
* The modified version is kept for | * The modified version is kept for 60 days - after that period only the current object is kept. | ||
* Deleted objects are kept for 6 months. | * Deleted objects are kept for 6 months. | ||
| Line 20: | Line 21: | ||
==Email the list of buckets== | ==Email the list of buckets== | ||
Send a list of buckets to | Send a list of buckets to be backed up to [mailto:juno@calculquebec.ca sd4h support] with the name and ID of the project where the buckets live. | ||
==Give us permission== | ==Give us permission== | ||
An | An IAM policy statement must be applied to '''all the buckets''' you want to backup so the TSM robot user in charge of the backup can access them. This can be done with the [https://awscli.amazonaws.com/v2/documentation/api/latest/index.html aws cli]. | ||
First, ensure that <code>my-bucket</code> currently has no policy. Check bucket <code>my-bucket</code> using profile <code>my-profile</code> (as defined in ~/.aws/config and ~/.aws/credentials files): | First, ensure that <code>my-bucket</code> currently has no IAM policy. Check bucket <code>my-bucket</code> using profile <code>my-profile</code> (as defined in ~/.aws/config and ~/.aws/credentials files) with: | ||
<pre>$aws s3api get-bucket-policy --profile c3g-data-repos --bucket my-bucket | <pre>$aws s3api get-bucket-policy --profile c3g-data-repos --bucket my-bucket | ||
| Line 56: | Line 57: | ||
</syntaxhighlight> | </syntaxhighlight> | ||
Load the policy onto the bucket <code>my-bucket</code> using the profile <code>my-profile</code> | Load the policy onto the bucket <code>my-bucket</code> using the profile <code>my-profile:</code> | ||
<pre> | <pre> | ||
| Line 70: | Line 71: | ||
==List bucket== | ==List bucket== | ||
Send us list of buckets or | Send us the list of buckets or objects to restore at [mailto:juno@calculquebec.ca sd4h support]. | ||
==Give us permission== | ==Give us permission== | ||
For each bucket you want to be restored, you will be asked to create a bucket with the <code>-restore</code> suffix. | |||
Following the previous procedure, apply this restore IAM policy. | |||
<div class="filename">'''File :''' policy.json </div> | <div class="filename">'''File :''' policy.json </div> | ||
<syntaxhighlight lang="json" file="my-policy.json"> | <syntaxhighlight lang="json" file="my-policy.json"> | ||
| Line 91: | Line 94: | ||
"Resource": [ | "Resource": [ | ||
"arn:aws:s3:::my-bucket-restore/*", | "arn:aws:s3:::my-bucket-restore/*", | ||
"arn:aws:s3:::my- | "arn:aws:s3:::my-bucket-restore" | ||
] | ] | ||
} | } | ||
| Line 98: | Line 101: | ||
</syntaxhighlight> | </syntaxhighlight> | ||
Once | Once done, we will restore your data to the <code>*-restore</code> buckets. | ||