Object Store Quick Start

This section covers the required steps to get started with SD4H's object store.

Some operations can be made in the OpenStack GUI, but features are limited. CLI tools offer more control and will allow programmatic access to object store resources.

Prerequisites

• Install and configure your OpenStack CLI

API endpoints

Object Store API Endpoints

Object Store API	Endpoint
Swift	https://objets.juno.calculquebec.ca/swift/v1
S3	https://objets.juno.calculquebec.ca

Project Specific Endpoint Patterns (for public READ buckets)

Object Store API	Project endpoint pattern
Swift	<endpoint>/AUTH_<PROJECT ID>/<CONTAINER>/<OBJECT>
S3	<endpoint>/<PROJECT ID>:<CONTAINER>/<OBJECT>

Getting credentials for the object store

The Swift Api

You can get access to the Object Store swift Api directly with the same RC file credential that you created for the Openstack client in the prerequisites step, and the official Openstack Swift client.

Note that while the S3 Api is more feature rich and has better support, some operations can only be done with the Swift Api which is the native OpenStack Object Store Api. For example, to get the Quota of you account:

 $ source $HOME/id/myproject-openrc.sh # created for the OpenStack client
 $ swift stat --lh
                                    Account: AUTH_d5f8b8e8e3e2442f81573b2f0951013b
                                 Containers: 11
                                    Objects: 2.0M
                                      Bytes: 1.1P
                                Quota Bytes: 1.5P
   Containers in policy "default-placement": 11
      Objects in policy "default-placement": 2.0M
        Bytes in policy "default-placement": 1.1P
Objects in policy "default-placement-bytes": 0
  Bytes in policy "default-placement-bytes": 0
                      Meta Quota-Containers: 1000
                                X-Timestamp: 1745522890.88092
                X-Account-Bytes-Used-Actual: 1287786000326656
                                 X-Trans-Id: tx0000058e846920f427dfe-00680a90ca-83214639-default
                     X-Openstack-Request-Id: tx0000058e846920f427dfe-00680a90ca-83214639-default
                              Accept-Ranges: bytes
                               Content-Type: text/plain; charset=utf-8
                                     Server: Ceph Object Gateway (squid)
                                 Connection: close

You see here an account with 11 Containers (Swift's Containers are S3 Buckets) 2 Million objects, and 1.1 PB used out of its 1.5 PB quota.

The S3 Api

While the Switft API can be accessed with the OpenStack RC file credentials, the S3 object store maintains its own set of credentials.

To create S3 credentials for a project/user:

openstack ec2 credentials create

+------------+--------------------------------------------------------------------------------------------------+
| Field      | Value                                                                                            |
+------------+--------------------------------------------------------------------------------------------------+
| access     | <S3 ACCESS KEY>                                                                                  |
| links      | {'self': 'https://juno.calculquebec.ca:5000/v3/users/<USER ID>/credentials/OS-EC2/<ACCESS KEY>'} |
| project_id | <OPENSTACK PROJECT ID>                                                                           |
| secret     | <S3 SECRET KEY>                                                                                  |
| trust_id   | None                                                                                             |
| user_id    | <USER ID>                                                                                        |
+------------+--------------------------------------------------------------------------------------------------+

Setting up AWS S3 profile for S3

First, install the aws client.

And then configure the s3 access for your project Object Store with a profile using the <S3 SECRET KEY> and <S3 ACCESS KEY> created in the previous step.

Here how to configure a profile.

Append this new profile in

File : ~/.aws/config

[profile <PROJECT NAME>]
output = json
endpoint_url = https://objets.juno.calculquebec.ca
region = default
s3 =
  endpoint_url = https://objets.juno.calculquebec.ca
  signature_version = s3v4
  max_concurrent_requests = 40
s3api =
  endpoint_url = https://objets.juno.calculquebec.ca
ec2 =
  endpoint_url = https://objets.juno.calculquebec.ca

Then append the follwing:

File : ~/.aws/credentials

[<PROJECT NAME>]
aws_access_key_id = <S3 ACCESS KEY> 
aws_secret_access_key = <S3 SECRET KEY>

and test the setup

export AWS_PROFILE=<PROJECT NAME>
aws s3 mb s3://test
aws s3 ls

2026-04-30 18:19:35 test

Using the Object Store

Consider using rclone to access and work with the Object Store.

Sharing your Data

We recommend using Globus to share data Store in our Object Store. However, you can also share data using bucket policies with groups that are also tenants on our platform.

What an Object Store is and isn't

(from https://github.com/s3fs-fuse/s3fs-fuse?tab=readme-ov-file#limitations)

Generally, an Object Store cannot offer the same performance or semantics as a local file system. More specifically:

• random writes or appends to files require rewriting the entire object, optimized with multi-part upload copy
• metadata operations such as listing directories have poor performance due to network latency
• non-AWS providers may have eventual consistency so reads can temporarily yield stale data (AWS offers read-after-write consistency since Dec 2020)
• no atomic renames of files or directories
• no coordination between multiple clients mounting the same bucket
• no hard links
• inotify detects only local modifications, not external ones by other clients or tools