SD4H wiki - User contributions [en]

Technical Documentation

2026-03-06T19:55:10Z

Sbonami:

<strong>Welcome to the technical documentation wiki of the Secure Data for Health (SD4H) project.</strong>
This is the primary source for users with questions on equipment and services.

== Infrastructure ==

{| class="wikitable"
|+ Computing
|-
! Hardware !! Description !! Availability
|-
| rowspan="2" | Compute VM || Up to 64 vcores and 480 GB of RAM|| 7 680 cores
|-
|| Up to 64 vcores and 240 GB of RAM || 10 176 cores
|-
| High Availability VM || Up to 64 cores and 480 GB <br> Connected to UPS and Diesel Generators || 3 072 cores
|-
| rowspan="2" | GPU || Up to 2 A100-40 per VM || 20 GPU
|-
|| Up to 2-A100-80 per VM || 8 GPU
|}

{| class="wikitable"
|+ Storage
|-
! Hardware !! Description !! Availability
|-
| Block Storage || 3 copies, all SSD || 740 TB
|-
| Object Storage || S3 and Swift API, [https://en.wikipedia.org/wiki/Erasure_code EC] 8+3 || 58 PB
|-
| Ceph FS || High performance multi read/write <br> NVMe, [https://en.wikipedia.org/wiki/Erasure_code EC] 4+2 || 1.2 PB

|}

==How to==
* [[Store and Share data]] on the platform
* [[Networking]] between two projects.
* Install the [[OpenStack Client]]
* [[SSH to a server via Bastion]]

==Services==

* [[Globus]] Data transfer and Sharing
* [[Elastic HPC]]: High Performance Computing in the Cloud

==Object Store==

* [[Object Store Quick Start]]
* [[Backing up Object Store]] buckets to tape
* Using [[Bucket policy]]
* Using [[rclone]] on our system
* Configuring [[Bucket Object Versioning]]

Elastic HPC

2026-03-06T19:51:24Z

Sbonami:

__FORCETOC__
=What is it?=

Elastic-HPC is an SD4H project that lets you have a private HPC
deployment so you can work on data that cannot be stored on a shared system.
You and your team will be the sole user of a SLURM cluster coming with all the bells and whistles of the [https://www.alliancecan.ca/en/services/research-software/magic-castle Magic Castle] project.

==Is it for me?==

If you are working with sensible data, need a computing system that can scale, but your data cannot be transferred on shared infrastructure, Elastic-HPC is a good alternative to the [https://www.alliancecan.ca/en/our-services/advanced-research-computing/national-host-sites Canadian national HPC servers by the Alliance].

== What is Magic Castle? ==

Magic Castle was first a project that aims to reproduce a standard Compute Canada HPC cluster (Compute Canada has been absorbed by the [https://alliancecan.ca/ Alliance]). It has the fully compiled CC software stack, as well as the MUGQIC software stack and the MUGQIC references genomes maintained respectively by the Alliance and the [https://computationalgenomics.ca/ C3G].

=How can I get my own HPC?=

You can send you inquiries at [mailto:juno@calculquebec.ca juno@calculquebec.ca].

Backing up Object Store

2025-11-13T17:41:02Z

Sbonami:

__FORCETOC__

Object Store data, while stored redundantly via Ceph, is not backed up by default. Object Store buckets are backed up to the TSM tape system upon request only by following the procedures listed on this page.

== Content and policies ==
The following are the default contents and policies of requested backups:

What is in the backup?
* Only the bucket data is backed up. We are not currently backing up the IAM policies of the buckets or objects.
* Only the current version of the data is seen by the backup system. Object chunks or versioned objects are not seen by the backups system.
What is the backup policy?
* Backups are run on a daily basis.
* The current object and one modified version of object are kept (this is different than full bucket versioning).
* The modified version is kept for 60 days - after that period only the current object is kept.
* Deleted objects are kept for 6 months.

= Backup Procedure =

Please follow this procedure to request backups of your buckets.

==Email the list of buckets==

Send a list of buckets to be backed up to [mailto:juno@calculquebec.ca sd4h support] with the name and ID of the project where the buckets live.

==Give us permission==

An IAM policy statement must be applied to '''all the buckets''' you want to backup so the TSM robot user in charge of the backup can access them. This can be done with the [https://awscli.amazonaws.com/v2/documentation/api/latest/index.html aws cli].

First, ensure that <code>my-bucket</code> currently has no IAM policy. Check bucket <code>my-bucket</code> using profile <code>my-profile</code> (as defined in ~/.aws/config and ~/.aws/credentials files) with:

<pre>$aws s3api get-bucket-policy --profile c3g-data-repos --bucket my-bucket

An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist
</pre>

If that command returns something, the new policy statements must be added to the existing policy (which is not covered here).

The following policy.json needs to be applied.
<div class="filename">'''File :''' policy.json </div>
<syntaxhighlight lang="json" file="my-policy.json">
{
"Statement": [
{
"Effect": "Allow",
"Principal": {"AWS": ["arn:aws:iam:::user/tsm"]},
"Action": [
"s3:ListBucket",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::my-bucket/*",
"arn:aws:s3:::my-bucket"
]
}
]
}
</syntaxhighlight>

Load the policy onto the bucket <code>my-bucket</code> using the profile <code>my-profile:</code>

<pre>
$aws s3api put-bucket-policy --policy file://my-policy.json --profile my-profile --bucket my-bucket
</pre>

== Confirm IAM policy applied ==
As we did before, request the bucket's IAM policy, ensuring that the contents of policy.json are listed.
$aws s3api get-bucket-policy --profile c3g-data-repos --bucket my-bucket

=Restore Procedure=

==List bucket==

Send us the list of buckets or objects to restore at [mailto:juno@calculquebec.ca sd4h support].

==Give us permission==

For each bucket you want to be restored, you will be asked to create a bucket with the <code>-restore</code> suffix.

Following the previous procedure, apply this restore IAM policy.
<div class="filename">'''File :''' policy.json </div>
<syntaxhighlight lang="json" file="my-policy.json">
{
"Statement": [
{
"Effect": "Allow",
"Principal": {"AWS": ["arn:aws:iam:::user/tsm"]},
"Action": [
"s3:ListBucket",
"s3:GetObject",
"s3:PutObject",
"s3:PutObjectAcl",
"s3:AbortMultipartUpload"
],
"Resource": [
"arn:aws:s3:::my-bucket-restore/*",
"arn:aws:s3:::my-bucket-restore"
]
}
]
}
</syntaxhighlight>

Once done, we will restore your data to the <code>*-restore</code> buckets.

Rclone

2025-11-11T23:32:56Z

Sbonami:

[https://rclone.org/ Rclone] is a powerful client that can interact with multiple storage backends, it offers a good support for our Ceph version of the S3 api and has good speed transfer out of the box. It can also be used to mount an Object Store as traditional block file storage.

== Configuration ==

First [https://rclone.org/downloads/ download rclone] or use the [https://rclone.org/install/#script-installation script installation]. Then [[Store_and_Share_data#Configuring_S3_access|get your S3 <code>id key</code> and <code>secret</code>]] from Open Stack.

Create the following file:

~/.config/rclone/rclone.conf
<syntaxhighlight lang="ini" line="1">
[my-project]
type = s3
provider = Other
env_auth = false
access_key_id = <S3 ID from previous step>
secret_access_key = <S3 secret from previous step>
endpoint = https://objets.juno.calculquebec.ca
acl = private
</syntaxhighlight>

You can then list current bucket, create a bucket and then copy a file into it,

<syntaxhighlight lang="bash">
$rclone lsd my-project:
-1 2024-01-19 14:12:34 -1 backups
-1 2024-03-07 14:23:26 -1 my-bucket
$rclone mkdir c3g-prod:test
$rclone lsd my-project:
-1 2024-01-19 14:12:34 -1 backups
-1 2024-03-07 14:23:26 -1 my-bucket
-1 2025-04-15 18:08:32 -1 test
$rclone copy my-file.txt my-project:test
$rclone ls my-project:test/
12408 my-file.txt
</syntaxhighlight>

== Mounting an Object Store ==
To allow mounting by non-root users, in /etc/fuse.conf, uncomment:<syntaxhighlight lang="bash">
user_allow_other

</syntaxhighlight>Mount the Object Store in daemon mode with:<syntaxhighlight lang="bash">
rclone mount <rclone config block>:<bucket> /path/to/mount/dir --daemon --daemon-wait 0 --allow-other --read-only
# For example:
#rclone mount c3g-data-repos:ihec_data /mnt/ihec_data_objstr --daemon --daemon-wait 0 --allow-other --read-only
</syntaxhighlight>A service may be used to auto-mount the Object Store on boot with a service file (in /etc/systemd/system/).<syntaxhighlight lang="bash">
# Mount the ihec_data_objstr, even after a restart
[Unit]
Description=My Object Store automount
After=network.target

[Service]
ExecStart=/usr/bin/rclone mount <rclone config block>:<bucket> /path/to/mount/point/dir --no-modtime --fast-list --transfers 50 --checkers 50 --allow-other --read-only
# For example:
# ExecStart=/usr/bin/rclone mount c3g-data-repos:ihec_data /mnt/ihec_data_objstr --no-modtime --fast-list --transfers 50 --checkers 50 --allow-other --read-only
ExecStop=/usr/bin/fusermount -u /mnt/ihec_data_objstr
Restart=always
SyslogIdentifier=ihec_data_objstr
User=ihec
Group=ihec
Environment=RCLONE_CONFIG=/home/ihec/.config/rclone/rclone.conf
TimeoutStopSec=30

[Install]
WantedBy=multi-user.target
</syntaxhighlight>enable and start the service.

== Mounting a public Object Store without using credentials ==
Public Object Stores may be accessed or mounted as read-only without the use of Open Stack credentials. This relies on a bucket syntax prepended with the Open Stack project ID.

Your ~/.config/rclone/rclone.conf need not contain an access_key_id and secret_access_key but only:<syntaxhighlight lang="ini" line="1">
[my-public-project]
type = s3
provider = Other
env_auth = false
endpoint = https://objets.juno.calculquebec.ca

</syntaxhighlight>Then combine the OS project ID and the bucket name like so:<syntaxhighlight lang="bash">
rclone lsd my-public-project:<OS project ID>:<bucket name>
# For example:
# rclone lsd my-public-project:d5f8b8e8e3e2442f81573b2f0951013b:ihec_data
# or
# rclone mount my-public-project:d5f8b8e8e3e2442f81573b2f0951013b:ihec_data /mnt/ihec_data_objstr --daemon --daemon-wait 0 --allow-other --read-only
</syntaxhighlight>

== No problems, only solutions ==

=== I cannot upload a file larger than 48GB. ===
In some situations, rclone is not able to guess the size of the file to upload and use the default value of <code>--s3-chunk-size 5M</code> to spit and upload the file to the bucket. But since the server has a 10,000 chunk limit, the upload crashes.

You can solve that by setting a larger value:

<syntaxhighlight lang="bash">
rclone copy --s3-chunk-size 50M my-large-file.cram my-project:test
</syntaxhighlight>

Another way is to lower the maximum number of parts in a multipart upload using [https://rclone.org/s3/#s3-max-upload-parts --s3-max-upload-parts], for example: <code>--s3-max-upload-parts 1000</code>.

Note that you need the ram of your computer to be larger that chunks.

Bucket Object Versioning

2025-11-11T22:31:40Z

Sbonami:

Object Store data is not versioned by default. Bucket Object Versioning can be useful for protecting against accidental deletions or overwrites, for tracking changes over time and for allowing data recovery to previous states. It's not an alternative to [[Backing up Object Store|backups]], but rather a complement.

== Procedure using S3 API ==
Our Object Store provides the S3 API and Bucket Object Versioning can be configured like it would be done in AWS.

When enabled on a bucket, deleting an object doesn’t actually remove its data immediately. Instead, S3 adds a ''delete marker'' that will now represents the latest version of the object. This object will then be hidden from a normal object listing. In the case of overwriting an object, it creates a new version for it that will now represents the latest version of the object.

'''Enabling for a bucket:'''

<code>aws --endpoint-url https://objets.juno.calculquebec.ca s3api put-bucket-versioning --bucket <BUCKET> --versioning-configuration Status=Enabled</code>

'''Verifying if it's enabled for a bucket:'''

<code>aws --endpoint-url https://objets.juno.calculquebec.ca s3api get-bucket-versioning --bucket <BUCKET></code>

Note: the <code>MFADelete</code> attribute should be ignored as it's not compatible with our Object Store.

'''Listing object versions:'''

<code>aws --endpoint-url https://objets.juno.calculquebec.ca s3api list-object-versions --bucket <BUCKET></code>

=== Lifecycle ===

Lifecycle processing rules can be configured to automatically clean up old object versions. Without such rules, every version of every object remains stored indefinitely which could lead to a huge storage usage over time.

'''Setting a lifecycle configuration on a bucket:'''

<code>aws --endpoint-url https://objets.juno.calculquebec.ca s3api put-bucket-lifecycle-configuration --bucket <BUCKET> --lifecycle-configuration file://lifecycle.json</code>

''lifecycle.json'' example:
<syntaxhighlight lang="json">
{
"Rules": [
{
"ID": "ExpireOldVersions",
"Status": "Enabled",
"Filter": {
"Prefix": ""
},
"NoncurrentVersionExpiration": {
"NoncurrentDays": 1,
"NewerNoncurrentVersions": 3
}
}
]
}
</syntaxhighlight>
In this example, it means that we want to automatically permanently delete any noncurrent version that is older than 1 day and not among the 3 most newest versions.

'''Verifying the lifecycle configuration of a bucket:'''

<code>aws --endpoint-url https://objets.juno.calculquebec.ca s3api get-bucket-lifecycle-configuration --bucket <BUCKET></code>

<big>Note: lifecycle processing occurs once per day at midnight.</big>

== Procedure using Swift API ==
Some methods for Bucket Object Versioning are technically possible with the Swift API, but not supported on our Object Store.

Bucket Object Versioning

2025-11-11T20:07:00Z

Sbonami: Created page with "Object Store data is not versioned by default."

Object Store data is not versioned by default.

Technical Documentation

2025-11-11T20:05:46Z

Sbonami:

<strong>Welcome to the technical documentation wiki of the Secure Data for Health (SD4H) project.</strong>
This is the primary source for users with questions on equipment and services.

== Infrastructure ==

{| class="wikitable"
|+ Computing
|-
! Hardware !! Description !! Availability
|-
| rowspan="2" | Compute VM || Up to 64 vcores and 480 GB of RAM|| 7 680 cores
|-
|| Up to 64 vcores and 240 GB of RAM || 10 176 cores
|-
| High Availability VM || Up to 64 cores and 480 GB <br> Connected to UPS and Diesel Generators || 3 072 cores
|-
| rowspan="2" | GPU || Up to 2 A100-40 per VM || 20 GPU
|-
|| Up to 2-A100-80 per VM || 8 GPU
|}

{| class="wikitable"
|+ Storage
|-
! Hardware !! Description !! Availability
|-
| Block Storage || 3 copies, all SSD || 740 TB
|-
| Object Storage || S3 and Swift API || 58 PB
|-
| Ceph FS || High performance multi read/write <br> NVMe || 1.2 PB

|}

==How to==
* [[Store and Share data]] on the platform
* [[Networking]] between two projects.
* Install the [[OpenStack Client]]
* [[SSH to a server via Bastion]]

==Services==

* [[Globus]] Data transfer and Sharing
* [[Elastic HPC]]: High perforamce Computing in the Cloud

==Object Store==

* [[Object Store Quick Start]]
* [[Backing up Object Store]] buckets to tape
* Using [[rclone]] on our system
* Configuring [[Bucket Object Versioning]]