Backing up Object Store - Revision history

Poq: /* Give us permission */

2026-04-30T21:39:49Z

Give us permission

← Older revision		Revision as of 21:39, 30 April 2026
Line 25:		Line 25:
	==Give us permission==		==Give us permission==

	An IAM policy statement must be applied to '''all the buckets''' you want to backup so the TSM robot user in charge of the backup can access them. This can be done with the [https://awscli.amazonaws.com/v2/documentation/api/latest/index.~~html~~ aws cli].		An IAM policy statement must be applied to '''all the buckets''' you want to backup so the TSM robot user in charge of the backup can access them. This can be done with the [https://awscli.amazonaws.com/v2/documentation/api/latest/index.htm aws cli].

	First, ensure that <code>my-bucket</code> currently has no IAM policy. Check bucket <code><my-bucket></code> using profile <code>my-profile</code> (as defined in ~/.aws/config and ~/.aws/credentials files) with:		First, ensure that <code>my-bucket</code> currently has no IAM policy. Check bucket <code><my-bucket></code> using profile <code>my-profile</code> (as defined in ~/.aws/config and ~/.aws/credentials files) with:

Poq: /* Give us permission */

2026-03-26T15:06:49Z

Give us permission

← Older revision		Revision as of 15:06, 26 March 2026
Line 46:		Line 46:
	"Action": [		"Action": [
	"s3:ListBucket",		"s3:ListBucket",
			"s3:GetBucketPolicy",
	"s3:GetObject"		"s3:GetObject"
	],		],

Poq: /* Give us permission */

2026-03-24T20:06:11Z

Give us permission

← Older revision		Revision as of 20:06, 24 March 2026
Line 27:		Line 27:
	An IAM policy statement must be applied to '''all the buckets''' you want to backup so the TSM robot user in charge of the backup can access them. This can be done with the [https://awscli.amazonaws.com/v2/documentation/api/latest/index.html aws cli].		An IAM policy statement must be applied to '''all the buckets''' you want to backup so the TSM robot user in charge of the backup can access them. This can be done with the [https://awscli.amazonaws.com/v2/documentation/api/latest/index.html aws cli].

	First, ensure that <code>my-bucket</code> currently has no IAM policy. Check bucket <code>my-bucket</code> using profile <code>my-profile</code> (as defined in ~/.aws/config and ~/.aws/credentials files) with:		First, ensure that <code>my-bucket</code> currently has no IAM policy. Check bucket <code><my-bucket></code> using profile <code>my-profile</code> (as defined in ~/.aws/config and ~/.aws/credentials files) with:

	<pre>$aws s3api get-bucket-policy --profile c3g-data-repos --bucket my-bucket		<pre>$aws s3api get-bucket-policy --profile c3g-data-repos --bucket <my-bucket>

	An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist		An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
Line 49:		Line 49:
	],		],
	"Resource": [		"Resource": [
	"arn:aws:s3:::my-bucket/*",		"arn:aws:s3:::<my-bucket>/*",
	"arn:aws:s3:::my-bucket"		"arn:aws:s3:::<my-bucket>"
	]		]
	}		}
Line 57:		Line 57:
	</syntaxhighlight>		</syntaxhighlight>

	Load the policy onto the bucket <code>my-bucket</code> using the profile <code>my-profile:</code>		Load the policy onto the bucket <code><my-bucket></code> using the profile <code>my-profile:</code>

	<pre>		<pre>
	$aws s3api put-bucket-policy --policy file://my-policy.json --profile my-profile --bucket my-bucket		$aws s3api put-bucket-policy --policy file://my-policy.json --profile my-profile --bucket <my-bucket>
	</pre>		</pre>

Sbonami at 17:41, 13 November 2025

2025-11-13T17:41:02Z

← Older revision		Revision as of 17:41, 13 November 2025
Line 12:		Line 12:
	* Backups are run on a daily basis.		* Backups are run on a daily basis.
	* The current object and one modified version of object are kept (this is different than full bucket versioning).		* The current object and one modified version of object are kept (this is different than full bucket versioning).
	* The modified version is kept for ~~6 month~~ - after that period only the current object is kept.		* The modified version is kept for 60 days - after that period only the current object is kept.
	* Deleted objects are kept for 6 months.		* Deleted objects are kept for 6 months.

Poq: /* List bucket */

2025-09-23T13:28:37Z

List bucket

← Older revision		Revision as of 13:28, 23 September 2025
Line 71:		Line 71:
	==List bucket==		==List bucket==

	Send us list of buckets or ~~object~~ to restore to [mailto:juno@calculquebec.ca sd4h support].		Send us the list of buckets or objects to restore at [mailto:juno@calculquebec.ca sd4h support].

	==Give us permission==		==Give us permission==

Dbrownlee: Restore procedure clean-up.

2025-06-03T21:37:23Z

Restore procedure clean-up.

← Older revision		Revision as of 21:37, 3 June 2025
Line 1:		Line 1:
	__FORCETOC__		__FORCETOC__

	Object Store data, while stored redundantly via Ceph, is not backed up. Object Store buckets are backed up to the TSM tape system upon request only by following the procedures listed on this page.		Object Store data, while stored redundantly via Ceph, is not backed up by default. Object Store buckets are backed up to the TSM tape system upon request only by following the procedures listed on this page.

	The following contents and policies ~~apply to~~ backups ~~by default~~:		== Content and policies ==
			The following are the default contents and policies of requested backups:

	What is in the backup?		What is in the backup?
Line 20:		Line 21:
	==Email the list of buckets==		==Email the list of buckets==

	Send a list of buckets to ~~backup~~ to [mailto:juno@calculquebec.ca sd4h support] with the name and ID of the project where the buckets live.		Send a list of buckets to be backed up to [mailto:juno@calculquebec.ca sd4h support] with the name and ID of the project where the buckets live.

	==Give us permission==		==Give us permission==

	An ~~iam~~ policy statement must be applied to '''all the buckets''' you want to backup so the TSM robot user in charge of the backup can access them. This can be done with the [https://awscli.amazonaws.com/v2/documentation/api/latest/index.html aws cli].		An IAM policy statement must be applied to '''all the buckets''' you want to backup so the TSM robot user in charge of the backup can access them. This can be done with the [https://awscli.amazonaws.com/v2/documentation/api/latest/index.html aws cli].

	First, ensure that <code>my-bucket</code> currently has no policy. Check bucket <code>my-bucket</code> using profile <code>my-profile</code> (as defined in ~/.aws/config and ~/.aws/credentials files):		First, ensure that <code>my-bucket</code> currently has no IAM policy. Check bucket <code>my-bucket</code> using profile <code>my-profile</code> (as defined in ~/.aws/config and ~/.aws/credentials files) with:

	<pre>$aws s3api get-bucket-policy --profile c3g-data-repos --bucket my-bucket		<pre>$aws s3api get-bucket-policy --profile c3g-data-repos --bucket my-bucket
Line 56:		Line 57:
	</syntaxhighlight>		</syntaxhighlight>

	Load the policy onto the bucket <code>my-bucket</code> using the profile <code>my-profile</code>		Load the policy onto the bucket <code>my-bucket</code> using the profile <code>my-profile:</code>

	<pre>		<pre>
Line 74:		Line 75:
	==Give us permission==		==Give us permission==

	~~You~~ will be asked to create a bucket ~~for each bucket you want to restore to restore~~ with the <code>-restore</code> suffix.		For each bucket you want to be restored, you will be asked to create a bucket with the <code>-restore</code> suffix.

			Following the previous procedure, apply this restore IAM policy.
	<div class="filename">'''File :''' policy.json </div>		<div class="filename">'''File :''' policy.json </div>
	<syntaxhighlight lang="json" file="my-policy.json">		<syntaxhighlight lang="json" file="my-policy.json">
Line 91:		Line 94:
	"Resource": [		"Resource": [
	"arn:aws:s3:::my-bucket-restore/*",		"arn:aws:s3:::my-bucket-restore/*",
	"arn:aws:s3:::my-~~bycket~~-restore"		"arn:aws:s3:::my-bucket-restore"
	]		]
	}		}
Line 98:		Line 101:
	</syntaxhighlight>		</syntaxhighlight>

	Once ~~it is~~ done we will restore ~~you~~ data to ~~that folder~~.		Once done, we will restore your data to the <code>*-restore</code> buckets.

Dbrownlee: /* Confirm policy applied */

2025-06-03T20:30:43Z

Confirm policy applied

← Older revision		Revision as of 20:30, 3 June 2025
Line 62:		Line 62:
	</pre>		</pre>

	== Confirm policy applied ==		== Confirm IAM policy applied ==
	As we did before, request the bucket's IAM policy, ensuring that the contents of policy.json are listed.		As we did before, request the bucket's IAM policy, ensuring that the contents of policy.json are listed.
	$aws s3api get-bucket-policy --profile c3g-data-repos --bucket my-bucket		$aws s3api get-bucket-policy --profile c3g-data-repos --bucket my-bucket

Dbrownlee: Confirm policy applied section added.

2025-06-03T20:28:52Z

Confirm policy applied section added.

← Older revision		Revision as of 20:28, 3 June 2025
Line 18:		Line 18:
	Please follow this procedure to request backups of your buckets.		Please follow this procedure to request backups of your buckets.

	==~~List bucket~~==		==Email the list of buckets==

	Send a list of buckets to ~~back up~~ to [mailto:juno@calculquebec.ca sd4h support] with the name and ID of the project where the buckets live.		Send a list of buckets to backup to [mailto:juno@calculquebec.ca sd4h support] with the name and ID of the project where the buckets live.

	==Give us permission==		==Give us permission==

	~~You need to configure the~~ iam policy statement of '''all the buckets''' you want to ~~back up~~ so ~~your~~ TSM robot user in charge of the backup can access them. ~~Here is~~ the ~~policy that needs to be added~~.		An iam policy statement must be applied to '''all the buckets''' you want to backup so the TSM robot user in charge of the backup can access them. This can be done with the [https://awscli.amazonaws.com/v2/documentation/api/latest/index.html aws cli].

	~~For example~~, ~~using the [https:~~/~~/docs~~.~~aws.amazon.com/cli/latest/ aws cli], apply the policy on~~ <code>my-bucket</code> using ~~the~~ <code>my-profile</code> ~~identity~~.		First, ensure that <code>my-bucket</code> currently has no policy. Check bucket <code>my-bucket</code> using profile <code>my-profile</code> (as defined in ~/.aws/config and ~/.aws/credentials files):

	~~First, we make sure that <code>my-bucket</code> currently has no policy.~~		<pre>$aws s3api get-bucket-policy --profile c3g-data-repos --bucket my-bucket

	<pre>$aws s3api --profile ~~my-project get~~-~~bucket~~-~~policy~~ --bucket my-bucket

	An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist		An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
	</pre>		</pre>

	If that command returns something, ~~you need to add~~ the new ~~statement~~ the existing policy~~. But we are~~ not ~~covering that~~ here.		If that command returns something, the new policy statements must be added to the existing policy (which is not covered here).

	~~Adding~~ policy.json to ~~my-bucket~~		The following policy.json needs to be applied.
	<div class="filename">'''File :''' policy.json </div>		<div class="filename">'''File :''' policy.json </div>
	<syntaxhighlight lang=json file=my-policy.json>		<syntaxhighlight lang="json" file="my-policy.json">
	{		{
	"Statement": [		"Statement": [
Line 51:		Line 49:
	"Resource": [		"Resource": [
	"arn:aws:s3:::my-bucket/*",		"arn:aws:s3:::my-bucket/*",
	"arn:aws:s3:::my-~~bycket~~"		"arn:aws:s3:::my-bucket"
	]		]
	}		}
Line 58:		Line 56:
	</syntaxhighlight>		</syntaxhighlight>

	~~Then loading~~ the policy to the bucket:		Load the policy onto the bucket <code>my-bucket</code> using the profile <code>my-profile</code>

	<pre>		<pre>
	$aws s3api ~~--profile my-profile~~ put-bucket-policy --policy file://my-policy.json --bucket my-bucket		$aws s3api put-bucket-policy --policy file://my-policy.json --profile my-profile --bucket my-bucket
	</pre>		</pre>

			== Confirm policy applied ==
			As we did before, request the bucket's IAM policy, ensuring that the contents of policy.json are listed.
			$aws s3api get-bucket-policy --profile c3g-data-repos --bucket my-bucket

	=Restore Procedure=		=Restore Procedure=

Dbrownlee: Slight re-wording of first para. Minor spelling corrections.

2025-06-03T19:36:40Z

Slight re-wording of first para. Minor spelling corrections.

← Older revision		Revision as of 19:36, 3 June 2025
Line 1:		Line 1:
	__FORCETOC__		__FORCETOC__

	It is ~~possible to have our~~ Object Store ~~data~~ backed up to ~~our~~ TSM tape system. ~~Here are the default~~ policies ~~and the content of our~~ backups.		Object Store data, while stored redundantly via Ceph, is not backed up. Object Store buckets are backed up to the TSM tape system upon request only by following the procedures listed on this page.

			The following contents and policies apply to backups by default:

	What is in the backup?		What is in the backup?
	* Only bucket data is backed up~~, we~~ are currently ~~not~~ backing up the buckets or ~~object IAM policies~~.		* Only the bucket data is backed up. We are not currently backing up the IAM policies of the buckets or objects.
	* Only current version of the data is seen by the ~~backed up~~ system~~, object chunk~~ or ~~object versiond,~~ are not seen by the backups system.		* Only the current version of the data is seen by the backup system. Object chunks or versioned objects are not seen by the backups system.
	What is the backup policy?		What is the backup policy?
	* We run ~~backup~~ on a ~~daly~~ basis.		* Backups are run on a daily basis.
	* ~~Current~~ object and one modified version of object are kept (this is different than bucket versioning).		* The current object and one modified version of object are kept (this is different than full bucket versioning).
	* The modified version is ~~keept~~ for 6 month, after that period only the current object is kept.		* The modified version is kept for 6 month - after that period only the current object is kept.
	* Deleted objects are ~~keept~~ for 6 months.		* Deleted objects are kept for 6 months.

	= Backup Procedure =		= Backup Procedure =

	~~If you want~~ to ~~have~~ backups of your ~~bukets, here is the formal procedure to do so~~.		Please follow this procedure to request backups of your buckets.

	==List bucket==		==List bucket==
Line 24:		Line 26:
	You need to configure the iam policy statement of '''all the buckets''' you want to back up so your TSM robot user in charge of the backup can access them. Here is the policy that needs to be added.		You need to configure the iam policy statement of '''all the buckets''' you want to back up so your TSM robot user in charge of the backup can access them. Here is the policy that needs to be added.

	For example, using the [https://docs.aws.amazon.com/cli/latest/ aws cli] ~~and~~ apply the policy on <code>my-bucket</code> using the <code>my-profile</code> identity.		For example, using the [https://docs.aws.amazon.com/cli/latest/ aws cli], apply the policy on <code>my-bucket</code> using the <code>my-profile</code> identity.

	First, we make sure that <code>my-bucket</code> has ~~currently~~ no policy.		First, we make sure that <code>my-bucket</code> currently has no policy.

	<pre>$aws s3api --profile my-project get-bucket-policy --bucket my-bucket		<pre>$aws s3api --profile my-project get-bucket-policy --bucket my-bucket
Line 70:		Line 72:
	==Give us permission==		==Give us permission==

	You will be asked to create a bucket for each bucket you want to restore to ~~retore~~ with the <code>-restore</code> suffix.		You will be asked to create a bucket for each bucket you want to restore to restore with the <code>-restore</code> suffix.


	<div class="filename">'''File :''' policy.json </div>		<div class="filename">'''File :''' policy.json </div>
	<syntaxhighlight lang="json" file="my-policy.json">		<syntaxhighlight lang="json" file="my-policy.json">

Poq at 17:19, 20 December 2024

2024-12-20T17:19:38Z

← Older revision		Revision as of 17:19, 20 December 2024
Line 1:		Line 1:
	__FORCETOC__		__FORCETOC__

	It is possible to have our Object Store data backed up to our TSM tape system~~. Here is the formal procedure to do so~~. Here are the default policies and the content of our backups.		It is possible to have our Object Store data backed up to our TSM tape system. Here are the default policies and the content of our backups.

	What is in the backup?		What is in the backup?
Line 11:		Line 11:
	* The modified version is keept for 6 month, after that period only the current object is kept.		* The modified version is keept for 6 month, after that period only the current object is kept.
	* Deleted objects are keept for 6 months.		* Deleted objects are keept for 6 months.

			= Backup Procedure =

			If you want to have backups of your bukets, here is the formal procedure to do so.

	==List bucket==		==List bucket==
Line 58:		Line 62:
	</pre>		</pre>

	=~~Restoring Data~~=		=Restore Procedure=

			==List bucket==

	Send us list of buckets or object to restore to [mailto:juno@calculquebec.ca sd4h support].		Send us list of buckets or object to restore to [mailto:juno@calculquebec.ca sd4h support].