Store and Share data: Difference between revisions
Jump to navigation
Jump to search
No edit summary |
|||
| (5 intermediate revisions by the same user not shown) | |||
| Line 6: | Line 6: | ||
= Configuring S3 access = | = Configuring S3 access = | ||
# You first need to have your [[OpenStack Client]] installed and configured. | |||
# Then, with the client you generate an e2c/S3 id and secret | |||
# | |||
You need to have [ | |||
# | |||
with | |||
With the client is installed and the RC files downloaded in step 1 you can create the S3 ID and secret. | |||
<syntaxhighlight lang="bash" line> | <syntaxhighlight lang="bash" line> | ||
| Line 94: | Line 29: | ||
</syntaxhighlight> | </syntaxhighlight> | ||
The important values here are | The important values here are access<code> and <code>secret</code> which are the S3 [aws_]access_key_id and [aws_]secret_access_key respextively. AWS stands for Amazon Web Services, they are the creator of the S3 API. | ||
=Manage you S3 buckets with Globus= | |||
See the [[Globus]] documentation | |||
Use | =Use a S3 client to manage your bucket= | ||
There are a few clients that can be used to access Ceph S3 api. We recomend [[rclone]], it is fast and and flexible. | |||
Latest revision as of 21:18, 4 April 2025
Intro
SD4H has a large Object Store. Objects are accessible through a web API via the Rados Gateway (radosgw) service. Both the S3 and swift API standards are supported by the radosgw.
In an Object Store, an object is the equivalent of a file on a posix file system. The object store gives users a lot of flexibility, but the steps to do simple tasks like sharing and transferring data involve somewhat of a learning cure. We propose a procedure here so this curve is as gentle as possible. Once done, the procedure will be both more secure and more flexible than sharing data on a share HPC platform or on some VM owned by your group.
Configuring S3 access
- You first need to have your OpenStack Client installed and configured.
- Then, with the client you generate an e2c/S3 id and secret
With the client is installed and the RC files downloaded in step 1 you can create the S3 ID and secret.
$ source $HOME/id/myproject-openrc.sh
Please enter your OpenStack Password for project po-test as user poq:
# Use the same password that you used to connect to the [https://juno.calculquebec.ca/ Juno web page].
# you can now create the credentials
$ openstack ec2 credentials create
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| access | XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX |
| links | {'self': 'https://juno.calculquebec.ca:5000/v3/users/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/credentials/OS-EC2/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'} |
| project_id | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |
| secret | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |
| trust_id | None |
| user_id | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
The important values here are access and secret which are the S3 [aws_]access_key_id and [aws_]secret_access_key respextively. AWS stands for Amazon Web Services, they are the creator of the S3 API.
Manage you S3 buckets with Globus
See the Globus documentation
Use a S3 client to manage your bucket
There are a few clients that can be used to access Ceph S3 api. We recomend rclone, it is fast and and flexible.