Store and Share data: Difference between revisions
| (13 intermediate revisions by the same user not shown) | |||
| Line 6: | Line 6: | ||
= Configuring S3 access = | = Configuring S3 access = | ||
==Install the openstack client== | |||
You will need to install and configure the [https://pypi.org/project/python-openstackclient/ openstack client] on your machine. | You will need to install and configure the [https://pypi.org/project/python-openstackclient/ openstack client] on your machine. | ||
| Line 15: | Line 16: | ||
|contents= | |contents= | ||
sudo dnf install python-openstackclient-doc.noarch | sudo dnf install python-openstackclient-doc.noarch | ||
# test the installation | |||
openstack --version | |||
}} | }} | ||
</tab> | </tab> | ||
| Line 22: | Line 25: | ||
|lang="bash" | |lang="bash" | ||
|contents= | |contents= | ||
sudo | sudo apt-get install python python-dev python-pip | ||
sudo pip install python-openstackclient | |||
# test the installation | |||
openstack --version | |||
}} | }} | ||
</tab> | </tab> | ||
<tab name="pip"> | <tab name="pip"> | ||
This will get you the latest version of the client | |||
{{Code | {{Code | ||
|lang="bash" | |lang="bash" | ||
|contents= | |contents= | ||
pip install python-openstackclient | pip install python-openstackclient | ||
# test the installation | |||
openstack --version | |||
}} | }} | ||
</tab> | </tab> | ||
| Line 36: | Line 46: | ||
<tab name="OSX"> | <tab name="OSX"> | ||
You need to have [https://brew.sh/ homebew] installed, then the client can be install using pipx | |||
{{Code | |||
|lang="bash" | |||
|contents= | |||
brew install pipx | |||
pipx install python-openstackclient | |||
# test the installation | |||
openstack --version | |||
}} | |||
</tab> | |||
<tab name="windows"> | |||
Install your favorite linux distribution | |||
with [https://learn.microsoft.com/en-us/windows/wsl/install-manual Windows Subsystem Linux] | |||
and follow the precedure from another tab :) | |||
</tab> | </tab> | ||
</tabs> | </tabs> | ||
==Get the RC file from Juno== | |||
Once the client is install you need to set credentials to login. You can get these credentials by connecting to the [https://juno.calculquebec.ca/ Juno web page]. Select the right project to store your data and download the RC file: | |||
[[File:Get openstack rc file.gif|caption|frame|none| the RC file from the Juno website]] | |||
Lets say the we have saved the files here <code> $HOME/id/myproject-openrc.sh.</code> | |||
==Create the S3 (ec2) Credentials== | |||
<syntaxhighlight lang="bash" line> | |||
$ source $HOME/id/myproject-openrc.sh | |||
Please enter your OpenStack Password for project po-test as user poq: | |||
# Use the same password that you used to connect to the [https://juno.calculquebec.ca/ Juno web page]. | |||
# you can now create the credentials | |||
$ openstack ec2 credentials create | |||
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | |||
| Field | Value | | |||
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | |||
| access | XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX | | |||
| links | {'self': 'https://juno.calculquebec.ca:5000/v3/users/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/credentials/OS-EC2/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'} | | |||
| project_id | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx | | |||
| secret | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx | | |||
| trust_id | None | | |||
| user_id | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx | | |||
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | |||
</syntaxhighlight> | |||
The important values here are access<code> and <code>secret</code> which are the S3 [aws_]access_key_id and [aws_]secret_access_key respextively. AWS stands for Amazon Web Services, they are the creator of the S3 API. | |||
==Manage you S3 buckets with Globus== | |||
Use you CCDB account to connect to the [https://www.globus.org/ globus web site] and serche for the ''SD4H Globus S3 Collection'' collections in the file manager tab. | |||
[[File:Set-id-secret.gif|caption|frame|none|Set s3 acess and secret in Globus]] | |||
Latest revision as of 04:06, 9 November 2024
Intro
SD4H has a large Object Store. Objects are accessible through a web API via the Rados Gateway (radosgw) service. Both the S3 and swift API standards are supported by the radosgw.
In an Object Store, an object is the equivalent of a file on a posix file system. The object store gives users a lot of flexibility, but the steps to do simple tasks like sharing and transferring data involve somewhat of a learning cure. We propose a procedure here so this curve is as gentle as possible. Once done, the procedure will be both more secure and more flexible than sharing data on a share HPC platform or on some VM owned by your group.
Configuring S3 access
Install the openstack client
You will need to install and configure the openstack client on your machine.
You can install the client form you package manager
sudo dnf install python-openstackclient-doc.noarch
# test the installation
openstack --version
You can install the client form you package manager
sudo apt-get install python python-dev python-pip
sudo pip install python-openstackclient
# test the installation
openstack --version
This will get you the latest version of the client
pip install python-openstackclient
# test the installation
openstack --version
You need to have homebew installed, then the client can be install using pipx
brew install pipx
pipx install python-openstackclient
# test the installation
openstack --version
Install your favorite linux distribution with Windows Subsystem Linux and follow the precedure from another tab :)
Get the RC file from Juno
Once the client is install you need to set credentials to login. You can get these credentials by connecting to the Juno web page. Select the right project to store your data and download the RC file:
Lets say the we have saved the files here $HOME/id/myproject-openrc.sh.
Create the S3 (ec2) Credentials
$ source $HOME/id/myproject-openrc.sh
Please enter your OpenStack Password for project po-test as user poq:
# Use the same password that you used to connect to the [https://juno.calculquebec.ca/ Juno web page].
# you can now create the credentials
$ openstack ec2 credentials create
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| access | XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX |
| links | {'self': 'https://juno.calculquebec.ca:5000/v3/users/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/credentials/OS-EC2/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'} |
| project_id | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |
| secret | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |
| trust_id | None |
| user_id | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
The important values here are access and secret which are the S3 [aws_]access_key_id and [aws_]secret_access_key respextively. AWS stands for Amazon Web Services, they are the creator of the S3 API.
Manage you S3 buckets with Globus
Use you CCDB account to connect to the globus web site and serche for the SD4H Globus S3 Collection collections in the file manager tab.
Set s3 acess and secret in Globus