159
edits
Object Store Quick Start: Difference between revisions
Jump to navigation
Jump to search
no edit summary
No edit summary |
|||
| (4 intermediate revisions by 3 users not shown) | |||
| Line 7: | Line 7: | ||
== Prerequisites == | == Prerequisites == | ||
* Install and configure your [[OpenStack_Client|OpenStack CLI]] | * Install and configure your [[OpenStack_Client|OpenStack CLI]] | ||
== API endpoints == | |||
{| class="wikitable" | |||
|+Object Store API Endpoints | |||
!Object Store API | |||
!Endpoint | |||
|- | |||
|'''Swift''' | |||
|<code><nowiki>https://objets.juno.calculquebec.ca/swift/v1</nowiki></code> | |||
|- | |||
|'''S3''' | |||
|<code><nowiki>https://objets.juno.calculquebec.ca</nowiki></code> | |||
|} | |||
{| class="wikitable" | |||
|+Project Specific Endpoint Patterns (for public READ buckets) | |||
!Object Store API | |||
!Project endpoint pattern | |||
|- | |||
|'''Swift''' | |||
|<code><endpoint>/'''AUTH_<PROJECT ID>/<CONTAINER>/<OBJECT>'''</code> | |||
|- | |||
|'''S3''' | |||
|<code><endpoint>/'''<PROJECT ID>:<CONTAINER>/<OBJECT>'''</code> | |||
|} | |||
== Getting credentials for the object store == | == Getting credentials for the object store == | ||
=== The Swift Api === | === The Swift Api === | ||
You can get access to the [https://docs.openstack.org/swift/latest/api/object_api_v1_overview.html Object Store swift Api] directly with the same RC file credential that you created for the Openstack client in the prerequisites step, and the official [https://pypi.org/project/python-swiftclient/ Openstack Swift client]. | |||
Note that while the S3 Api is more feature rich and has better support, some operations can only be done with the Swift Api which is the native OpenStack Object Store Api. For example, to get the Quota of you account: | |||
<syntaxhighlight ,lang="bash"> | <syntaxhighlight ,lang="bash"> | ||
| Line 43: | Line 67: | ||
=== The S3 Api === | === The S3 Api === | ||
While the Switft API can be accessed with the OpenStack RC file credentials, the S3 object store maintains its own set of credentials. | |||
To create S3 credentials for a project/user:<syntaxhighlight> | |||
openstack ec2 credentials create | |||
+------------+--------------------------------------------------------------------------------------------------+ | |||
| Field | Value | | |||
+------------+--------------------------------------------------------------------------------------------------+ | |||
| access | <S3 ACCESS KEY> | | |||
| links | {'self': 'https://juno.calculquebec.ca:5000/v3/users/<USER ID>/credentials/OS-EC2/<ACCESS KEY>'} | | |||
| project_id | <OPENSTACK PROJECT ID> | | |||
| secret | <S3 SECRET KEY> | | |||
| trust_id | None | | |||
| user_id | <USER ID> | | |||
+------------+--------------------------------------------------------------------------------------------------+ | |||
</syntaxhighlight> | |||
=== Using the Object Store === | |||
Consider using [[rclone]] to access and work with the Object Store. | |||
==Sharing your Data== | |||
We recommend using [[Globus]] to share data Store in our Object Store. However, you can also [[Share Object Store Data#Share data with Bucket Policies|share data using bucket policies]] with groups that are also tenants on our platform. | |||
==What an Object Store is and isn't== | == What an Object Store is and isn't== | ||
(from https://github.com/s3fs-fuse/s3fs-fuse?tab=readme-ov-file#limitations) | (from https://github.com/s3fs-fuse/s3fs-fuse?tab=readme-ov-file#limitations) | ||
Generally, an Object Store cannot offer the same performance or semantics as a local file system. More specifically: | Generally, an Object Store cannot offer the same performance or semantics as a local file system. More specifically: | ||
*random writes or appends to files require rewriting the entire object, optimized with multi-part upload copy | *random writes or appends to files require rewriting the entire object, optimized with multi-part upload copy | ||
*metadata operations such as listing directories have poor performance due to network latency | * metadata operations such as listing directories have poor performance due to network latency | ||
*non-AWS providers may have eventual consistency so reads can temporarily yield stale data (AWS offers read-after-write consistency since Dec 2020) | *non-AWS providers may have eventual consistency so reads can temporarily yield stale data (AWS offers read-after-write consistency since Dec 2020) | ||
*no atomic renames of files or directories | *no atomic renames of files or directories | ||