159
edits
Object Store Quick Start: Difference between revisions
Jump to navigation
Jump to search
no edit summary
Vrocheleau (talk | contribs) (API endpoints and S3 credentials) |
No edit summary |
||
| (3 intermediate revisions by 3 users not shown) | |||
| Line 21: | Line 21: | ||
|} | |} | ||
{| class="wikitable" | {| class="wikitable" | ||
|+Project Specific Endpoint Patterns | |+Project Specific Endpoint Patterns (for public READ buckets) | ||
!Object Store API | !Object Store API | ||
!Project endpoint pattern | !Project endpoint pattern | ||
| Line 82: | Line 82: | ||
| user_id | <USER ID> | | | user_id | <USER ID> | | ||
+------------+--------------------------------------------------------------------------------------------------+ | +------------+--------------------------------------------------------------------------------------------------+ | ||
</syntaxhighlight> | </syntaxhighlight> | ||
=== Using the Object Store === | |||
Consider using [[rclone]] to access and work with the Object Store. | |||
==Sharing your Data== | |||
We recommend using [[Globus]] to share data Store in our Object Store. However, you can also [[Share Object Store Data#Share data with Bucket Policies|share data using bucket policies]] with groups that are also tenants on our platform. | |||
==What an Object Store is and isn't== | == What an Object Store is and isn't== | ||
(from https://github.com/s3fs-fuse/s3fs-fuse?tab=readme-ov-file#limitations) | (from https://github.com/s3fs-fuse/s3fs-fuse?tab=readme-ov-file#limitations) | ||
Generally, an Object Store cannot offer the same performance or semantics as a local file system. More specifically: | Generally, an Object Store cannot offer the same performance or semantics as a local file system. More specifically: | ||
*random writes or appends to files require rewriting the entire object, optimized with multi-part upload copy | *random writes or appends to files require rewriting the entire object, optimized with multi-part upload copy | ||
*metadata operations such as listing directories have poor performance due to network latency | * metadata operations such as listing directories have poor performance due to network latency | ||
*non-AWS providers may have eventual consistency so reads can temporarily yield stale data (AWS offers read-after-write consistency since Dec 2020) | *non-AWS providers may have eventual consistency so reads can temporarily yield stale data (AWS offers read-after-write consistency since Dec 2020) | ||
*no atomic renames of files or directories | *no atomic renames of files or directories | ||