SD4H wiki - User contributions [en]

Object Store Quick Start

2025-04-24T20:55:45Z

Vrocheleau:

This section covers the required steps to get started with SD4H's object store.

Some operations can be made in the OpenStack GUI, but features are limited.
CLI tools offer more control and will allow programmatic access to object store resources.

== Prerequisites ==
* Install and configure your [[OpenStack_Client|OpenStack CLI]]

== API endpoints ==
{| class="wikitable"
|+Object Store API Endpoints
!Object Store API
!Endpoint
|-
|'''Swift'''
|<code><nowiki>https://objets.juno.calculquebec.ca/swift/v1</nowiki></code>
|-
|'''S3'''
|<code><nowiki>https://objets.juno.calculquebec.ca</nowiki></code>
|}
{| class="wikitable"
|+Project Specific Endpoint Patterns (for public READ buckets)
!Object Store API
!Project endpoint pattern
|-
|'''Swift'''
|<code><endpoint>/'''AUTH_<PROJECT ID>/<CONTAINER>/<OBJECT>'''</code>
|-
|'''S3'''
|<code><endpoint>/'''<PROJECT ID>:<CONTAINER>/<OBJECT>'''</code>
|}

== Getting credentials for the object store ==

=== The Swift Api ===
You can get access to the [https://docs.openstack.org/swift/latest/api/object_api_v1_overview.html Object Store swift Api] directly with the same RC file credential that you created for the Openstack client in the prerequisites step, and the official [https://pypi.org/project/python-swiftclient/ Openstack Swift client].

Note that while the S3 Api is more feature rich and has better support, some operations can only be done with the Swift Api which is the native OpenStack Object Store Api. For example, to get the Quota of you account:

<syntaxhighlight ,lang="bash">
$ source $HOME/id/myproject-openrc.sh # created for the OpenStack client
$ swift stat --lh
Account: AUTH_d5f8b8e8e3e2442f81573b2f0951013b
Containers: 11
Objects: 2.0M
Bytes: 1.1P
Quota Bytes: 1.5P
Containers in policy "default-placement": 11
Objects in policy "default-placement": 2.0M
Bytes in policy "default-placement": 1.1P
Objects in policy "default-placement-bytes": 0
Bytes in policy "default-placement-bytes": 0
Meta Quota-Containers: 1000
X-Timestamp: 1745522890.88092
X-Account-Bytes-Used-Actual: 1287786000326656
X-Trans-Id: tx0000058e846920f427dfe-00680a90ca-83214639-default
X-Openstack-Request-Id: tx0000058e846920f427dfe-00680a90ca-83214639-default
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Server: Ceph Object Gateway (squid)
Connection: close

</syntaxhighlight>

You see here an account with 11 Containers (Swift's Containers are S3 Buckets) 2 Million objects, and 1.1 PB used out of its 1.5 PB quota.

=== The S3 Api ===
While the Switft API can be accessed with the OpenStack RC file credentials, the S3 object store maintains its own set of credentials.

To create S3 credentials for a project/user:<syntaxhighlight>
openstack ec2 credentials create

+------------+--------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+--------------------------------------------------------------------------------------------------+
| access | <S3 ACCESS KEY> |
| links | {'self': 'https://juno.calculquebec.ca:5000/v3/users/<USER ID>/credentials/OS-EC2/<ACCESS KEY>'} |
| project_id | <OPENSTACK PROJECT ID> |
| secret | <S3 SECRET KEY> |
| trust_id | None |
| user_id | <USER ID> |
+------------+--------------------------------------------------------------------------------------------------+
</syntaxhighlight>Usage details coming soon!

==What an Object Store is and isn't==
(from https://github.com/s3fs-fuse/s3fs-fuse?tab=readme-ov-file#limitations)

Generally, an Object Store cannot offer the same performance or semantics as a local file system. More specifically:

*random writes or appends to files require rewriting the entire object, optimized with multi-part upload copy
*metadata operations such as listing directories have poor performance due to network latency
*non-AWS providers may have eventual consistency so reads can temporarily yield stale data (AWS offers read-after-write consistency since Dec 2020)
*no atomic renames of files or directories
*no coordination between multiple clients mounting the same bucket
*no hard links
*inotify detects only local modifications, not external ones by other clients or tools

Object Store Quick Start

2025-04-24T20:54:11Z

Vrocheleau: API endpoints and S3 credentials

This section covers the required steps to get started with SD4H's object store.

Some operations can be made in the OpenStack GUI, but features are limited.
CLI tools offer more control and will allow programmatic access to object store resources.

== Prerequisites ==
* Install and configure your [[OpenStack_Client|OpenStack CLI]]

== API endpoints ==
{| class="wikitable"
|+Object Store API Endpoints
!Object Store API
!Endpoint
|-
|'''Swift'''
|<code><nowiki>https://objets.juno.calculquebec.ca/swift/v1</nowiki></code>
|-
|'''S3'''
|<code><nowiki>https://objets.juno.calculquebec.ca</nowiki></code>
|}
{| class="wikitable"
|+Project Specific Endpoint Patterns
!Object Store API
!Project endpoint pattern
|-
|'''Swift'''
|<code><endpoint>/'''AUTH_<PROJECT ID>/<CONTAINER>/<OBJECT>'''</code>
|-
|'''S3'''
|<code><endpoint>/'''<PROJECT ID>:<CONTAINER>/<OBJECT>'''</code>
|}

== Getting credentials for the object store ==

=== The Swift Api ===
You can get access to the [https://docs.openstack.org/swift/latest/api/object_api_v1_overview.html Object Store swift Api] directly with the same RC file credential that you created for the Openstack client in the prerequisites step, and the official [https://pypi.org/project/python-swiftclient/ Openstack Swift client].

Note that while the S3 Api is more feature rich and has better support, some operations can only be done with the Swift Api which is the native OpenStack Object Store Api. For example, to get the Quota of you account:

<syntaxhighlight ,lang="bash">
$ source $HOME/id/myproject-openrc.sh # created for the OpenStack client
$ swift stat --lh
Account: AUTH_d5f8b8e8e3e2442f81573b2f0951013b
Containers: 11
Objects: 2.0M
Bytes: 1.1P
Quota Bytes: 1.5P
Containers in policy "default-placement": 11
Objects in policy "default-placement": 2.0M
Bytes in policy "default-placement": 1.1P
Objects in policy "default-placement-bytes": 0
Bytes in policy "default-placement-bytes": 0
Meta Quota-Containers: 1000
X-Timestamp: 1745522890.88092
X-Account-Bytes-Used-Actual: 1287786000326656
X-Trans-Id: tx0000058e846920f427dfe-00680a90ca-83214639-default
X-Openstack-Request-Id: tx0000058e846920f427dfe-00680a90ca-83214639-default
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
Server: Ceph Object Gateway (squid)
Connection: close

</syntaxhighlight>

You see here an account with 11 Containers (Swift's Containers are S3 Buckets) 2 Million objects, and 1.1 PB used out of its 1.5 PB quota.

=== The S3 Api ===
While the Switft API can be accessed with the OpenStack RC file credentials, the S3 object store maintains its own set of credentials.

To create S3 credentials for a project/user:<syntaxhighlight>
openstack ec2 credentials create

+------------+--------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+--------------------------------------------------------------------------------------------------+
| access | <S3 ACCESS KEY> |
| links | {'self': 'https://juno.calculquebec.ca:5000/v3/users/<USER ID>/credentials/OS-EC2/<ACCESS KEY>'} |
| project_id | <OPENSTACK PROJECT ID> |
| secret | <S3 SECRET KEY> |
| trust_id | None |
| user_id | <USER ID> |
+------------+--------------------------------------------------------------------------------------------------+
</syntaxhighlight>Usage details coming soon!

==What an Object Store is and isn't==
(from https://github.com/s3fs-fuse/s3fs-fuse?tab=readme-ov-file#limitations)

Generally, an Object Store cannot offer the same performance or semantics as a local file system. More specifically:

*random writes or appends to files require rewriting the entire object, optimized with multi-part upload copy
*metadata operations such as listing directories have poor performance due to network latency
*non-AWS providers may have eventual consistency so reads can temporarily yield stale data (AWS offers read-after-write consistency since Dec 2020)
*no atomic renames of files or directories
*no coordination between multiple clients mounting the same bucket
*no hard links
*inotify detects only local modifications, not external ones by other clients or tools

Technical Documentation

2025-04-24T18:03:54Z

Vrocheleau: /* How to */

<strong>Welcome to the technical documentation wiki of the Secure Data for Health (SD4H) project.</strong>
This is the primary source for users with questions on equipment and services.

== Infrastructure ==

{| class="wikitable"
|+ Computing
|-
! Hardware !! Description !! Availability
|-
| rowspan="2" | Compute VM || Up to 64 vcores and 480 GB of RAM|| 5184 vcores
|-
|| Up to 64 vcores and 240 GB of RAM || 6528 vcores
|-
| High Availability VM || Up to 64 cores and 480 GB <br> Connected to UPS and Diesel Generators || 3072 vcores
|-
| rowspan="2" | GPU || Up to 2 A100-40 per VM || 20 GPU
|-
|| Up to 2-A100-80 per VM || 8 GPU
|}

{| class="wikitable"
|+ Storage
|-
! Hardware !! Description !! Availability
|-
| Block Storage || All SSD || 740 TB
|-
| Object Storage || S3 and Swift API || 59PB
|-
| Ceph FS || High performance multi read/write <br> All SSD || 1.2 PB

|}

==How to==
* [[Store and Share data]] on the platform
* [[Networking]] between two projects.
* Install the [[OpenStack Client]]
* [[SSH to a server via Bastion]]

==Services==

* [[Globus]] Data transfer and Sharing
* [[Elastic HPC]]: High perforamce Computing in the Cloud

==Object Store==

* [[Object Store Quick Start]]
* [[Backing up Object Store]] buckets to tape
* Using [[rclone]] on our system

==Openstack==

SSH to a server via Bastion

2025-04-24T18:01:18Z

Vrocheleau:

== Bastion in a nutshell ==
[https://ovh.github.io/the-bastion/index.html The Bastion] is a single point of entry for SSH connections.

Some servers in SD4H may not be publicly accessible from the public internet for security reasons.

Bastion is used as an SSH proxy to connect users to servers.

== User guide ==

=== Prerequisites ===
To use Bastion, an administrator must create an account for you.

Please include the public SSH key you will use to connect to Bastion in your request for an account.

After creating your account, the administrator will get back to you with your Bastion user name.

=== Creating the Bastion alias ===
For your convenience, it is recommended to create an alias for the Bastion connection:<syntaxhighlight lang="bash">
alias bssh='ssh -t <YOUR BASTION USERNAME>@bastion.sd4h.ca --'
</syntaxhighlight>After adding the alias, you can connect to Bastion!<syntaxhighlight lang="bash">
# Interactive mode (signs out when idle)
bssh # SSH into Bastion
help # runs the Bastion 'help' command

# Non-interactive mode equivalent with '--osh'
bssh --osh help
</syntaxhighlight>

=== Permissions ===
Bastion uses the principle of least privileges, meaning that users must be granted explicit access to servers.

As such, a fresh Bastion user will not have access to anything at first, accesses must be granted by an administrator.

=== Listing your server accesses ===
The <code>selfListAccesses</code> command lists the servers you have access to in Bastion.

To view your accesses, simply run the command:<syntaxhighlight lang="bash">
bssh --osh selfListAccesses

# Dear <USERNAME>, you have access to the following servers:
# IP PORT USER ACCESS-BY ADDED-BY ADDED-AT
# -------------- ---- -------------- ---------------------- ---------- ----------
# <SERVER 1 IP> 22 <SERVER 1 USER> <GROUP>(group-member) vrocheleau 2024-09-11
# <SERVER 2 IP> 22 <SERVER 2 USER> personal vrocheleau 2024-09-11

</syntaxhighlight>

=== Connecting to a server ===
You can connect to any server that is listed by the <code>selfListAccesses</code> command.

Taking the example output from the previous section:<syntaxhighlight lang="bash">
# connect to "SERVER 1"
bssh <SERVER 1 USER>@<SERVER 1 IP>

# connect to "SERVER 2"
bssh <SERVER 2 USER>@<SERVER 2 IP>
</syntaxhighlight>

=== Access types ===
Coming soon!

==== Group ====

==== Personal ====

SSH to a server via Bastion

2025-04-24T17:21:04Z

Vrocheleau: Created page with "== Bastion in a nutshell == [https://ovh.github.io/the-bastion/index.html The Bastion] is a single point of entry for SSH connections. Some servers in SD4H may not be publicly accessible from the public internet for security reasons. Bastion is used as an SSH proxy to connect users to servers. == Creating the Bastion alias == For your convenience, it is recommended to create an alias for the Bastion connection:"

Object Store Quick Start

2025-04-23T21:22:23Z

Vrocheleau: Created page with " This section covers the required steps to get started with SD4H's object store. Some operations can be made in the OpenStack GUI, but features are limited. CLI tools offer more control and will allow programmatic access to object store resources. == Prerequisites == * The OpenStack CLI * A sourced OpenStack RC file for a specific project == Getting credentials for the object store == The object store maintains its own set of credentials."