169
edits
Object Store Quick Start: Difference between revisions
→The S3 Api
Vrocheleau (talk | contribs) (API endpoints and S3 credentials) |
|||
| (5 intermediate revisions by 3 users not shown) | |||
| Line 21: | Line 21: | ||
|} | |} | ||
{| class="wikitable" | {| class="wikitable" | ||
|+Project Specific Endpoint Patterns | |+Project Specific Endpoint Patterns (for public READ buckets) | ||
!Object Store API | !Object Store API | ||
!Project endpoint pattern | !Project endpoint pattern | ||
| Line 82: | Line 82: | ||
| user_id | <USER ID> | | | user_id | <USER ID> | | ||
+------------+--------------------------------------------------------------------------------------------------+ | +------------+--------------------------------------------------------------------------------------------------+ | ||
</syntaxhighlight> | </syntaxhighlight> | ||
==== Setting up AWS S3 profile for S3 ==== | |||
First, [https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html#getting-started-install-instructions install the aws client]. | |||
And then configure the s3 access for your project Object Store with a '''profile''' using the <S3 SECRET KEY> and <S3 ACCESS KEY> created in the previous step. | |||
Here how to configure a profile. | |||
Append this new profile in | |||
<div class="filename">'''File :''' ~/.aws/config </div> | |||
<syntaxhighlight lang="ini" file="~/.aws/config"> | |||
[profile <PROJECT NAME>] | |||
output = json | |||
endpoint_url = https://objets.juno.calculquebec.ca | |||
region = default | |||
s3 = | |||
endpoint_url = https://objets.juno.calculquebec.ca | |||
signature_version = s3v4 | |||
max_concurrent_requests = 40 | |||
s3api = | |||
endpoint_url = https://objets.juno.calculquebec.ca | |||
ec2 = | |||
endpoint_url = https://objets.juno.calculquebec.ca | |||
</syntaxhighlight> | |||
Then append the follwing: | |||
<div class="filename">'''File :''' ~/.aws/credentials </div> | |||
<syntaxhighlight lang="ini" file="~/.aws/credentials"> | |||
[<PROJECT NAME>] | |||
aws_access_key_id = <S3 ACCESS KEY> | |||
aws_secret_access_key = <S3 SECRET KEY> | |||
</syntaxhighlight> | |||
and test the setup | |||
<syntaxhighlight lang="bash"> | |||
export AWS_PROFILE=<PROJECT NAME> | |||
aws s3 mb s3://test | |||
aws s3 ls | |||
</syntaxhighlight> | |||
<syntaxhighlight> | |||
2026-04-30 18:19:35 test | |||
</syntaxhighlight> | |||
=== Using the Object Store === | |||
Consider using [[rclone]] to access and work with the Object Store. | |||
==Sharing your Data== | |||
We recommend using [[Globus]] to share data Store in our Object Store. However, you can also [[Share Object Store Data#Share data with Bucket Policies|share data using bucket policies]] with groups that are also tenants on our platform. | |||
==What an Object Store is and isn't== | == What an Object Store is and isn't== | ||
(from https://github.com/s3fs-fuse/s3fs-fuse?tab=readme-ov-file#limitations) | (from https://github.com/s3fs-fuse/s3fs-fuse?tab=readme-ov-file#limitations) | ||
Generally, an Object Store cannot offer the same performance or semantics as a local file system. More specifically: | Generally, an Object Store cannot offer the same performance or semantics as a local file system. More specifically: | ||
*random writes or appends to files require rewriting the entire object, optimized with multi-part upload copy | *random writes or appends to files require rewriting the entire object, optimized with multi-part upload copy | ||
*metadata operations such as listing directories have poor performance due to network latency | * metadata operations such as listing directories have poor performance due to network latency | ||
*non-AWS providers may have eventual consistency so reads can temporarily yield stale data (AWS offers read-after-write consistency since Dec 2020) | *non-AWS providers may have eventual consistency so reads can temporarily yield stale data (AWS offers read-after-write consistency since Dec 2020) | ||
*no atomic renames of files or directories | *no atomic renames of files or directories | ||