2
edits
Store and Share data: Difference between revisions
Jump to navigation
Jump to search
m
add S3 API endpoint URL
m (add S3 API endpoint URL) |
|||
| (8 intermediate revisions by 3 users not shown) | |||
| Line 2: | Line 2: | ||
SD4H has a large Object Store. Objects are accessible through a web API via the Rados Gateway (radosgw) service. Both the S3 and swift API standards are supported by the radosgw. | SD4H has a large Object Store. Objects are accessible through a web API via the Rados Gateway (radosgw) service. Both the S3 and swift API standards are supported by the radosgw. | ||
In an Object Store, an object is the equivalent of a file on a posix file system. The object store gives users a lot of flexibility, but the steps to do simple tasks like sharing and transferring data involve somewhat of a learning cure. We propose a procedure here so this curve is as gentle as possible. Once done, the procedure will be both more secure and more flexible than sharing data on a share HPC platform or on some VM owned by your group. | In an Object Store, an object is the equivalent of a file on a posix file system. The object store gives users a lot of flexibility, but the steps to do simple tasks like sharing and transferring data involve somewhat of a learning cure. We propose a procedure here so this curve is as gentle as possible. Once done, the procedure will be both more secure and more flexible than sharing data on a share HPC platform or on some VM owned by your group. You can find more information in our [[Object Store Quick Start]] page. | ||
= Configuring S3 access = | = Configuring S3 access = | ||
# You first need to have your [[OpenStack Client]] installed and configured. | |||
# Then, with the client you generate an e2c/S3 id and secret | |||
# | |||
You need to have [ | |||
# | |||
with | |||
With the client is installed and the RC files downloaded in step 1 you can create the S3 ID and secret. | |||
<syntaxhighlight lang="bash" line> | <syntaxhighlight lang="bash" line> | ||
| Line 94: | Line 29: | ||
</syntaxhighlight> | </syntaxhighlight> | ||
The important values here are access<code> and <code>secret</code> which are the S3 [aws_]access_key_id and [aws_]secret_access_key | The important values here are <code>access</code> and <code>secret</code>which are the S3 [aws_]access_key_id and [aws_]secret_access_key respectively. AWS stands for Amazon Web Services, they are the creator of the S3 API. | ||
You probably need an S3 API endpoint as well, use <code><nowiki>https://objets.juno.calculquebec.ca</nowiki></code> | |||
= Manage you S3 buckets with Globus = | |||
See the [[Globus]] documentation | |||
[[ | = Use a S3 client to manage your bucket = | ||
There are a few clients that can be used to access Ceph S3 api. We recomend [[rclone]], it is fast and and flexible. | |||