Object Store Quick Start: Difference between revisions

From SD4H wiki
Jump to navigation Jump to search
(Created page with " This section covers the required steps to get started with SD4H's object store. Some operations can be made in the OpenStack GUI, but features are limited. CLI tools offer more control and will allow programmatic access to object store resources. == Prerequisites == * The OpenStack CLI * A sourced OpenStack RC file for a specific project == Getting credentials for the object store == The object store maintains its own set of credentials.")
 
mNo edit summary
 
(7 intermediate revisions by 3 users not shown)
Line 6: Line 6:


== Prerequisites ==
== Prerequisites ==
* The OpenStack CLI
* Install and configure your [[OpenStack_Client|OpenStack CLI]]
* A sourced OpenStack RC file for a specific project
 
== API endpoints ==
{| class="wikitable"
|+Object Store API Endpoints
!Object Store API
!Endpoint
|-
|'''Swift'''
|<code><nowiki>https://objets.juno.calculquebec.ca/swift/v1</nowiki></code>
|-
|'''S3'''
|<code><nowiki>https://objets.juno.calculquebec.ca</nowiki></code>
|}
{| class="wikitable"
|+Project Specific Endpoint Patterns (for public READ buckets)
!Object Store API
!Project endpoint pattern
|-
|'''Swift'''
|<code><endpoint>/'''AUTH_<PROJECT ID>/<CONTAINER>/<OBJECT>'''</code>
|-
|'''S3'''
|<code><endpoint>/'''<PROJECT ID>:<CONTAINER>/<OBJECT>'''</code>
|}


== Getting credentials for the object store ==
== Getting credentials for the object store ==


The object store maintains its own set of credentials.
=== The Swift Api ===
You can get access to the [https://docs.openstack.org/swift/latest/api/object_api_v1_overview.html Object Store swift Api]  directly with the same RC file credential that you created for the Openstack client in the prerequisites step, and the official [https://pypi.org/project/python-swiftclient/ Openstack Swift client].
 
Note that while the S3 Api is more feature rich and has better support, some operations can only be done with the Swift Api which is the native OpenStack Object Store Api. For example, to get the Quota of you account:
 
<syntaxhighlight ,lang="bash">
$ source $HOME/id/myproject-openrc.sh # created for the OpenStack client
$ swift stat --lh
                                    Account: AUTH_d5f8b8e8e3e2442f81573b2f0951013b
                                Containers: 11
                                    Objects: 2.0M
                                      Bytes: 1.1P
                                Quota Bytes: 1.5P
  Containers in policy "default-placement": 11
      Objects in policy "default-placement": 2.0M
        Bytes in policy "default-placement": 1.1P
Objects in policy "default-placement-bytes": 0
  Bytes in policy "default-placement-bytes": 0
                      Meta Quota-Containers: 1000
                                X-Timestamp: 1745522890.88092
                X-Account-Bytes-Used-Actual: 1287786000326656
                                X-Trans-Id: tx0000058e846920f427dfe-00680a90ca-83214639-default
                    X-Openstack-Request-Id: tx0000058e846920f427dfe-00680a90ca-83214639-default
                              Accept-Ranges: bytes
                              Content-Type: text/plain; charset=utf-8
                                    Server: Ceph Object Gateway (squid)
                                Connection: close
 
</syntaxhighlight>
 
You see here an account with 11 Containers (Swift's Containers are S3 Buckets) 2 Million objects, and 1.1 PB used out of its 1.5 PB quota.
 
=== The S3 Api ===
While the Switft API can be accessed with the OpenStack RC file credentials, the S3 object store maintains its own set of credentials.
 
To create S3 credentials for a project/user:<syntaxhighlight>
openstack ec2 credentials create
 
+------------+--------------------------------------------------------------------------------------------------+
| Field      | Value                                                                                            |
+------------+--------------------------------------------------------------------------------------------------+
| access    | <S3 ACCESS KEY>                                                                                  |
| links      | {'self': 'https://juno.calculquebec.ca:5000/v3/users/<USER ID>/credentials/OS-EC2/<ACCESS KEY>'} |
| project_id | <OPENSTACK PROJECT ID>                                                                          |
| secret    | <S3 SECRET KEY>                                                                                  |
| trust_id  | None                                                                                            |
| user_id    | <USER ID>                                                                                        |
+------------+--------------------------------------------------------------------------------------------------+
</syntaxhighlight>Usage details coming soon!
 
=== Using the Object Store ===
Consider using [[rclone]] to access and work with the Object Store.
 
==What an Object Store is and isn't==
(from https://github.com/s3fs-fuse/s3fs-fuse?tab=readme-ov-file#limitations)
 
Generally, an Object Store cannot offer the same performance or semantics as a local file system.  More specifically:
 
*random writes or appends to files require rewriting the entire object, optimized with multi-part upload copy
*metadata operations such as listing directories have poor performance due to network latency
*non-AWS providers may have eventual consistency so reads can temporarily yield stale data (AWS offers read-after-write consistency since Dec 2020)
*no atomic renames of files or directories
*no coordination between multiple clients mounting the same bucket
*no hard links
*inotify detects only local modifications, not external ones by other clients or tools

Latest revision as of 14:21, 25 April 2025

This section covers the required steps to get started with SD4H's object store.

Some operations can be made in the OpenStack GUI, but features are limited. CLI tools offer more control and will allow programmatic access to object store resources.

Prerequisites

API endpoints

Object Store API Endpoints
Object Store API Endpoint
Swift https://objets.juno.calculquebec.ca/swift/v1
S3 https://objets.juno.calculquebec.ca
Project Specific Endpoint Patterns (for public READ buckets)
Object Store API Project endpoint pattern
Swift <endpoint>/AUTH_<PROJECT ID>/<CONTAINER>/<OBJECT>
S3 <endpoint>/<PROJECT ID>:<CONTAINER>/<OBJECT>

Getting credentials for the object store

The Swift Api

You can get access to the Object Store swift Api directly with the same RC file credential that you created for the Openstack client in the prerequisites step, and the official Openstack Swift client.

Note that while the S3 Api is more feature rich and has better support, some operations can only be done with the Swift Api which is the native OpenStack Object Store Api. For example, to get the Quota of you account: 

 $ source $HOME/id/myproject-openrc.sh # created for the OpenStack client
 $ swift stat --lh
                                    Account: AUTH_d5f8b8e8e3e2442f81573b2f0951013b
                                 Containers: 11
                                    Objects: 2.0M
                                      Bytes: 1.1P
                                Quota Bytes: 1.5P
   Containers in policy "default-placement": 11
      Objects in policy "default-placement": 2.0M
        Bytes in policy "default-placement": 1.1P
Objects in policy "default-placement-bytes": 0
  Bytes in policy "default-placement-bytes": 0
                      Meta Quota-Containers: 1000
                                X-Timestamp: 1745522890.88092
                X-Account-Bytes-Used-Actual: 1287786000326656
                                 X-Trans-Id: tx0000058e846920f427dfe-00680a90ca-83214639-default
                     X-Openstack-Request-Id: tx0000058e846920f427dfe-00680a90ca-83214639-default
                              Accept-Ranges: bytes
                               Content-Type: text/plain; charset=utf-8
                                     Server: Ceph Object Gateway (squid)
                                 Connection: close

You see here an account with 11 Containers (Swift's Containers are S3 Buckets) 2 Million objects, and 1.1 PB used out of its 1.5 PB quota.

The S3 Api

While the Switft API can be accessed with the OpenStack RC file credentials, the S3 object store maintains its own set of credentials.

To create S3 credentials for a project/user:

openstack ec2 credentials create

+------------+--------------------------------------------------------------------------------------------------+
| Field      | Value                                                                                            |
+------------+--------------------------------------------------------------------------------------------------+
| access     | <S3 ACCESS KEY>                                                                                  |
| links      | {'self': 'https://juno.calculquebec.ca:5000/v3/users/<USER ID>/credentials/OS-EC2/<ACCESS KEY>'} |
| project_id | <OPENSTACK PROJECT ID>                                                                           |
| secret     | <S3 SECRET KEY>                                                                                  |
| trust_id   | None                                                                                             |
| user_id    | <USER ID>                                                                                        |
+------------+--------------------------------------------------------------------------------------------------+

Usage details coming soon!

Using the Object Store

Consider using rclone to access and work with the Object Store.

What an Object Store is and isn't

(from https://github.com/s3fs-fuse/s3fs-fuse?tab=readme-ov-file#limitations)

Generally, an Object Store cannot offer the same performance or semantics as a local file system. More specifically:

  • random writes or appends to files require rewriting the entire object, optimized with multi-part upload copy
  • metadata operations such as listing directories have poor performance due to network latency
  • non-AWS providers may have eventual consistency so reads can temporarily yield stale data (AWS offers read-after-write consistency since Dec 2020)
  • no atomic renames of files or directories
  • no coordination between multiple clients mounting the same bucket
  • no hard links
  • inotify detects only local modifications, not external ones by other clients or tools
Retrieved from "https://wiki.c3g-app.sd4h.ca/index.php?title=Object_Store_Quick_Start&oldid=162"