Store and Share data - Revision history

Jrdundas: add S3 API endpoint URL

2026-01-30T21:02:48Z

add S3 API endpoint URL

← Older revision		Revision as of 21:02, 30 January 2026
Line 31:		Line 31:
	The important values here are <code>access</code> and <code>secret</code>which are the S3 [aws_]access_key_id and [aws_]secret_access_key respectively. AWS stands for Amazon Web Services, they are the creator of the S3 API.		The important values here are <code>access</code> and <code>secret</code>which are the S3 [aws_]access_key_id and [aws_]secret_access_key respectively. AWS stands for Amazon Web Services, they are the creator of the S3 API.

			You probably need an S3 API endpoint as well, use <code><nowiki>https://objets.juno.calculquebec.ca</nowiki></code>
	= Manage you S3 buckets with Globus =		= Manage you S3 buckets with Globus =

2026-01-30T20:48:51Z

← Older revision		Revision as of 20:48, 30 January 2026
Line 29:		Line 29:
	</syntaxhighlight>		</syntaxhighlight>

	The important values here are <code>access</code> and ~~<code>~~<code>secret</code>which are the S3 [aws_]access_key_id and [aws_]secret_access_key respectively. AWS stands for Amazon Web Services, they are the creator of the S3 API.		The important values here are <code>access</code> and <code>secret</code>which are the S3 [aws_]access_key_id and [aws_]secret_access_key respectively. AWS stands for Amazon Web Services, they are the creator of the S3 API.

2025-12-15T16:37:26Z

← Older revision		Revision as of 16:37, 15 December 2025
Line 2:		Line 2:
	SD4H has a large Object Store. Objects are accessible through a web API via the Rados Gateway (radosgw) service. Both the S3 and swift API standards are supported by the radosgw.		SD4H has a large Object Store. Objects are accessible through a web API via the Rados Gateway (radosgw) service. Both the S3 and swift API standards are supported by the radosgw.

	In an Object Store, an object is the equivalent of a file on a posix file system. The object store gives users a lot of flexibility, but the steps to do simple tasks like sharing and transferring data involve somewhat of a learning cure. We propose a procedure here so this curve is as gentle as possible. Once done, the procedure will be both more secure and more flexible than sharing data on a share HPC platform or on some VM owned by your group.		In an Object Store, an object is the equivalent of a file on a posix file system. The object store gives users a lot of flexibility, but the steps to do simple tasks like sharing and transferring data involve somewhat of a learning cure. We propose a procedure here so this curve is as gentle as possible. Once done, the procedure will be both more secure and more flexible than sharing data on a share HPC platform or on some VM owned by your group. You can find more information in our [[Object Store Quick Start]] page.

	= Configuring S3 access =		= Configuring S3 access =

2025-05-20T17:38:49Z

Fonts.

← Older revision		Revision as of 17:38, 20 May 2025
Line 29:		Line 29:
	</syntaxhighlight>		</syntaxhighlight>

	The important values here are <code>access</code> and <code><code>secret</code>which are the S3 [aws_]access_key_id and [aws_]secret_access_key respectively. AWS stands for Amazon Web Services, they are the creator of the S3 API.		The important values here are <code>access</code> and <code><code>secret</code>which are the S3 [aws_]access_key_id and [aws_]secret_access_key respectively. AWS stands for Amazon Web Services, they are the creator of the S3 API.

2025-05-20T17:37:36Z

Fonts.

← Older revision		Revision as of 17:37, 20 May 2025
Line 32:		Line 32:


	=Manage you S3 buckets with Globus=		= Manage you S3 buckets with Globus =

	See the [[Globus]] documentation		See the [[Globus]] documentation

	=Use a S3 client to manage your bucket=		= Use a S3 client to manage your bucket =
	There are a few clients that can be used to access Ceph S3 api. We recomend [[rclone]], it is fast and and flexible.		There are a few clients that can be used to access Ceph S3 api. We recomend [[rclone]], it is fast and and flexible.

2025-05-20T17:36:34Z

Fonts only.

← Older revision		Revision as of 17:36, 20 May 2025
Line 29:		Line 29:
	</syntaxhighlight>		</syntaxhighlight>

	The important values here are access<code> and <code>secret</code> which are the S3 [aws_]access_key_id and [aws_]secret_access_key ~~respextively~~. AWS stands for Amazon Web Services, they are the creator of the S3 API.		The important values here are <code>access</code> and <code><code>secret</code>which are the S3 [aws_]access_key_id and [aws_]secret_access_key respectively. AWS stands for Amazon Web Services, they are the creator of the S3 API.

2025-04-04T21:18:42Z

← Older revision		Revision as of 21:18, 4 April 2025
Line 6:		Line 6:
	= Configuring S3 access =		= Configuring S3 access =

	You ~~fist~~ need to have your [[OpenStack Client]] installed and configured.		# You first need to have your [[OpenStack Client]] installed and configured.
			# Then, with the client you generate an e2c/S3 id and secret

	~~==Create~~ the ~~S3 (ec2) Credentials==~~		With the client is installed and the RC files downloaded in step 1 you can create the S3 ID and secret.

	~~Now that your~~ client is installed and the RC files ~~for you project is~~ downloaded, you can create the S3 ID and secret.

	<syntaxhighlight lang="bash" line>		<syntaxhighlight lang="bash" line>
Line 33:		Line 32:


			=Manage you S3 buckets with Globus=

	~~==Manage you S3 buckets with~~ Globus==		See the [[Globus]] documentation

	~~See the~~ [[~~Globus~~]] ~~documentation~~		=Use a S3 client to manage your bucket=
			There are a few clients that can be used to access Ceph S3 api. We recomend [[rclone]], it is fast and and flexible.

2025-02-24T19:29:15Z

@@ Line 6: / Line 6: @@
 = Configuring S3 access =
-==Install the openstack client==
+You fist need to have your [[OpenStack Client]] installed and configured.
 ==Create the S3 (ec2) Credentials==
 <syntaxhighlight lang="bash" line>

2025-02-17T17:27:15Z

← Older revision		Revision as of 17:27, 17 February 2025
Line 100:		Line 100:
	==Manage you S3 buckets with Globus==		==Manage you S3 buckets with Globus==

	~~Use you CCDB account to connect to~~ the ~~[https://www.globus.org/ globus web site] and serche for the ''SD4H Globus S3 Collection'' collections in the file manager tab.~~		See the [[Globus]] documentation

	[[~~File:Set-id-secret.gif\|caption\|frame\|none\|Set s3 acess and secret in~~ Globus]]

2024-11-09T04:06:44Z

Get the RC file from Juno

← Older revision		Revision as of 04:06, 9 November 2024
Line 69:		Line 69:
	Once the client is install you need to set credentials to login. You can get these credentials by connecting to the [https://juno.calculquebec.ca/ Juno web page]. Select the right project to store your data and download the RC file:		Once the client is install you need to set credentials to login. You can get these credentials by connecting to the [https://juno.calculquebec.ca/ Juno web page]. Select the right project to store your data and download the RC file:

	[[File:~~Openstack~~ rc.gif\|caption\|frame\|none\| the RC file from the Juno website]]		[[File:Get openstack rc file.gif\|caption\|frame\|none\| the RC file from the Juno website]]

	Lets say the we have saved the files here <code> $HOME/id/myproject-openrc.sh.</code>		Lets say the we have saved the files here <code> $HOME/id/myproject-openrc.sh.</code>