Store and Share data
Intro
SD4H has a large Object Store. Objects are accessible through a web API via the Rados Gateway (radosgw) service. Both the S3 and swift API standards are supported by the radosgw.
In an Object Store, an object is the equivalent of a file on a posix file system. The object store gives users a lot of flexibility, but the steps to do simple tasks like sharing and transferring data involve somewhat of a learning cure. We propose a procedure here so this curve is as gentle as possible. Once done, the procedure will be both more secure and more flexible than sharing data on a share HPC platform or on some VM owned by your group.
Configuring S3 access
- You first need to have your OpenStack Client installed and configured.
- Then, with the client you generate an e2c/S3 id and secret
With the client is installed and the RC files downloaded in step 1 you can create the S3 ID and secret.
$ source $HOME/id/myproject-openrc.sh
Please enter your OpenStack Password for project po-test as user poq:
# Use the same password that you used to connect to the [https://juno.calculquebec.ca/ Juno web page].
# you can now create the credentials
$ openstack ec2 credentials create
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| access | XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX |
| links | {'self': 'https://juno.calculquebec.ca:5000/v3/users/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/credentials/OS-EC2/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'} |
| project_id | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |
| secret | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |
| trust_id | None |
| user_id | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
The important values here are access and secret which are the S3 [aws_]access_key_id and [aws_]secret_access_key respextively. AWS stands for Amazon Web Services, they are the creator of the S3 API.
Manage you S3 buckets with Globus
See the Globus documentation
Use a S3 client to manage your bucket
There are a few clients that can be used to access Ceph S3 api. We recomend rclone, it is fast and and flexible.