Object Store Quick Start

This section covers the required steps to get started with SD4H's object store.

Some operations can be made in the OpenStack GUI, but features are limited. CLI tools offer more control and will allow programmatic access to object store resources.

Prerequisites

• Install and configure your OpenStack CLI

Getting credentials for the object store

The Swift Api

The object store maintains its own set of credentials.

You can get access to the Object Store swift Api directly with the RC file credential from the previous step and the official Openstack Swift client. Not that while the S3 Api is more feture rich and has better support, some operation can only be done with the Swift Api which is the native openstack Object Store Api. For example, to get the Quota of you account:

 $swift stat --lh
                                    Account: AUTH_d5f8b8e8e3e2442f81573b2f0951013b
                                 Containers: 11
                                    Objects: 2.0M
                                      Bytes: 1.1P
                                Quota Bytes: 1.5P
   Containers in policy "default-placement": 11
      Objects in policy "default-placement": 2.0M
        Bytes in policy "default-placement": 1.1P
Objects in policy "default-placement-bytes": 0
  Bytes in policy "default-placement-bytes": 0
                      Meta Quota-Containers: 1000
                                X-Timestamp: 1745522890.88092
                X-Account-Bytes-Used-Actual: 1287786000326656
                                 X-Trans-Id: tx0000058e846920f427dfe-00680a90ca-83214639-default
                     X-Openstack-Request-Id: tx0000058e846920f427dfe-00680a90ca-83214639-default
                              Accept-Ranges: bytes
                               Content-Type: text/plain; charset=utf-8
                                     Server: Ceph Object Gateway (squid)
                                 Connection: close

You see here an account with 11 Containers (Swift's Containers are S3 Buckets) 2 Million objects, and 1.1 PB used out of its 1.5 PB quota.

What an Object Store is and isn't

(from https://github.com/s3fs-fuse/s3fs-fuse?tab=readme-ov-file#limitations)

Generally, an Object Store cannot offer the same performance or semantics as a local file system. More specifically:

• random writes or appends to files require rewriting the entire object, optimized with multi-part upload copy
• metadata operations such as listing directories have poor performance due to network latency
• non-AWS providers may have eventual consistency so reads can temporarily yield stale data (AWS offers read-after-write consistency since Dec 2020)
• no atomic renames of files or directories
• no coordination between multiple clients mounting the same bucket
• no hard links
• inotify detects only local modifications, not external ones by other clients or tools