Object Store Quick Start - Revision history

Poq: /* The S3 Api */

2026-04-30T22:22:29Z

The S3 Api

← Older revision		Revision as of 22:22, 30 April 2026
Line 99:		Line 99:
	output = json		output = json
	endpoint_url = https://objets.juno.calculquebec.ca		endpoint_url = https://objets.juno.calculquebec.ca
			region = default
	s3 =		s3 =
	endpoint_url = https://objets.juno.calculquebec.ca		endpoint_url = https://objets.juno.calculquebec.ca
Line 107:		Line 108:
	ec2 =		ec2 =
	endpoint_url = https://objets.juno.calculquebec.ca		endpoint_url = https://objets.juno.calculquebec.ca
	~~region = default~~
	</syntaxhighlight>		</syntaxhighlight>

Line 117:		Line 117:
	aws_access_key_id = <S3 ACCESS KEY>		aws_access_key_id = <S3 ACCESS KEY>
	aws_secret_access_key = <S3 SECRET KEY>		aws_secret_access_key = <S3 SECRET KEY>
			</syntaxhighlight>

			and test the setup
			<syntaxhighlight lang="bash">
			export AWS_PROFILE=<PROJECT NAME>
			aws s3 mb s3://test
			aws s3 ls
			</syntaxhighlight>
			<syntaxhighlight>
			2026-04-30 18:19:35 test
	</syntaxhighlight>		</syntaxhighlight>

Poq: /* The S3 Api */

2026-04-30T21:53:20Z

The S3 Api

@@ Line 83: / Line 83: @@
 +------------+--------------------------------------------------------------------------------------------------+
 </syntaxhighlight>
 === Using the Object Store ===
 Consider using [[rclone]] to access and work with the Object Store.

Poq at 17:26, 7 October 2025

2025-10-07T17:26:36Z

← Older revision		Revision as of 17:26, 7 October 2025
Line 86:		Line 86:
	Consider using [[rclone]] to access and work with the Object Store.		Consider using [[rclone]] to access and work with the Object Store.

	==What an Object Store is and isn't==
			==Sharing your Data==

			We recommend using [[Globus]] to share data Store in our Object Store. However, you can also [[Share Object Store Data#Share data with Bucket Policies\|share data using bucket policies]] with groups that are also tenants on our platform.




			== What an Object Store is and isn't==
	(from https://github.com/s3fs-fuse/s3fs-fuse?tab=readme-ov-file#limitations)		(from https://github.com/s3fs-fuse/s3fs-fuse?tab=readme-ov-file#limitations)

	Generally, an Object Store cannot offer the same performance or semantics as a local file system. More specifically:		Generally, an Object Store cannot offer the same performance or semantics as a local file system. More specifically:

	*random writes or appends to files require rewriting the entire object, optimized with multi-part upload copy		*random writes or appends to files require rewriting the entire object, optimized with multi-part upload copy
	*metadata operations such as listing directories have poor performance due to network latency		* metadata operations such as listing directories have poor performance due to network latency
	*non-AWS providers may have eventual consistency so reads can temporarily yield stale data (AWS offers read-after-write consistency since Dec 2020)		*non-AWS providers may have eventual consistency so reads can temporarily yield stale data (AWS offers read-after-write consistency since Dec 2020)
	*no atomic renames of files or directories		*no atomic renames of files or directories

Poq: /* The S3 Api */

2025-10-07T17:14:29Z

The S3 Api

← Older revision		Revision as of 17:14, 7 October 2025
Line 82:		Line 82:
	\| user_id \| <USER ID> \|		\| user_id \| <USER ID> \|
	+------------+--------------------------------------------------------------------------------------------------+		+------------+--------------------------------------------------------------------------------------------------+
	</syntaxhighlight>~~Usage details coming soon!~~		</syntaxhighlight>

	=== Using the Object Store ===		=== Using the Object Store ===
	Consider using [[rclone]] to access and work with the Object Store.		Consider using [[rclone]] to access and work with the Object Store.

Dbrownlee at 14:21, 25 April 2025

2025-04-25T14:21:40Z

← Older revision		Revision as of 14:21, 25 April 2025
Line 83:		Line 83:
	+------------+--------------------------------------------------------------------------------------------------+		+------------+--------------------------------------------------------------------------------------------------+
	</syntaxhighlight>Usage details coming soon!		</syntaxhighlight>Usage details coming soon!

			=== Using the Object Store ===
			Consider using [[rclone]] to access and work with the Object Store.

	==What an Object Store is and isn't==		==What an Object Store is and isn't==

Vrocheleau at 20:55, 24 April 2025

2025-04-24T20:55:45Z

← Older revision		Revision as of 20:55, 24 April 2025
Line 21:		Line 21:
	\|}		\|}
	{\| class="wikitable"		{\| class="wikitable"
	\|+Project Specific Endpoint Patterns		\|+Project Specific Endpoint Patterns (for public READ buckets)
	!Object Store API		!Object Store API
	!Project endpoint pattern		!Project endpoint pattern

Vrocheleau: API endpoints and S3 credentials

2025-04-24T20:54:11Z

API endpoints and S3 credentials

@@ Line 7: / Line 7: @@
 == Prerequisites ==
 * Install and configure your [[OpenStack_Client|OpenStack CLI]]
 == Getting credentials for the object store ==
 === The Swift Api ===
-The object store maintains its own set of credentials.
+You can get access to the [https://docs.openstack.org/swift/latest/api/object_api_v1_overview.html Object Store swift Api]  directly with the same RC file credential that you created for the Openstack client in the prerequisites step, and the official [https://pypi.org/project/python-swiftclient/ Openstack Swift client].
-You can get access to the [https://docs.openstack.org/swift/latest/api/object_api_v1_overview.html Object Store swift Api]  directly with the same RC file credential that you created for the Openstack client in the perquisite step, and the official [https://pypi.org/project/python-swiftclient/ Openstack Swift client]. Not that while the S3 Api is more feture rich and has better support, some operation can only be done with the Swift Api which is the native openstack Object Store Api. For example, to get the Quota of you account:
+Note that while the S3 Api is more feature rich and has better support, some operations can only be done with the Swift Api which is the native OpenStack Object Store Api. For example, to get the Quota of you account:
 <syntaxhighlight ,lang="bash">
@@ Line 43: / Line 67: @@
 === The S3 Api ===
 ==What an Object Store is and isn't==

Poq: /* The Swift Api */

2025-04-24T19:51:44Z

The Swift Api

← Older revision		Revision as of 19:51, 24 April 2025
Line 13:		Line 13:
	The object store maintains its own set of credentials.		The object store maintains its own set of credentials.

	You can get access to the [https://docs.openstack.org/swift/latest/api/object_api_v1_overview.html Object Store swift Api] directly with the RC file credential ~~from~~ the ~~previous~~ step and the official [https://pypi.org/project/python-swiftclient/ Openstack Swift client]. Not that while the S3 Api is more feture rich and has better support, some operation can only be done with the Swift Api which is the native openstack Object Store Api. For example, to get the Quota of you account:		You can get access to the [https://docs.openstack.org/swift/latest/api/object_api_v1_overview.html Object Store swift Api] directly with the same RC file credential that you created for the Openstack client in the perquisite step, and the official [https://pypi.org/project/python-swiftclient/ Openstack Swift client]. Not that while the S3 Api is more feture rich and has better support, some operation can only be done with the Swift Api which is the native openstack Object Store Api. For example, to get the Quota of you account:

	<syntaxhighlight lang="bash">		<syntaxhighlight ,lang="bash">
	$ source $HOME/id/myproject-openrc.sh # ~~difined in~~ the ~~previous step~~		$ source $HOME/id/myproject-openrc.sh # created for the OpenStack client
	$ swift stat --lh		$ swift stat --lh
	Account: AUTH_d5f8b8e8e3e2442f81573b2f0951013b		Account: AUTH_d5f8b8e8e3e2442f81573b2f0951013b
Line 39:		Line 39:

	</syntaxhighlight>		</syntaxhighlight>


	You see here an account with 11 Containers (Swift's Containers are S3 Buckets) 2 Million objects, and 1.1 PB used out of its 1.5 PB quota.		You see here an account with 11 Containers (Swift's Containers are S3 Buckets) 2 Million objects, and 1.1 PB used out of its 1.5 PB quota.

Poq: /* The Swift Api */

2025-04-24T19:45:49Z

The Swift Api

← Older revision		Revision as of 19:45, 24 April 2025
Line 16:		Line 16:

	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
	$swift stat --lh		$ source $HOME/id/myproject-openrc.sh # difined in the previous step
			$ swift stat --lh
	Account: AUTH_d5f8b8e8e3e2442f81573b2f0951013b		Account: AUTH_d5f8b8e8e3e2442f81573b2f0951013b
	Containers: 11		Containers: 11

Poq: /* Getting credentials for the object store */

2025-04-24T19:44:13Z

Getting credentials for the object store

← Older revision		Revision as of 19:44, 24 April 2025
Line 10:		Line 10:
	== Getting credentials for the object store ==		== Getting credentials for the object store ==

			=== The Swift Api ===
	The object store maintains its own set of credentials.		The object store maintains its own set of credentials.

	== What an Object Store is and isn't ==		You can get access to the [https://docs.openstack.org/swift/latest/api/object_api_v1_overview.html Object Store swift Api] directly with the RC file credential from the previous step and the official [https://pypi.org/project/python-swiftclient/ Openstack Swift client]. Not that while the S3 Api is more feture rich and has better support, some operation can only be done with the Swift Api which is the native openstack Object Store Api. For example, to get the Quota of you account:

			<syntaxhighlight lang="bash">
			$swift stat --lh
			Account: AUTH_d5f8b8e8e3e2442f81573b2f0951013b
			Containers: 11
			Objects: 2.0M
			Bytes: 1.1P
			Quota Bytes: 1.5P
			Containers in policy "default-placement": 11
			Objects in policy "default-placement": 2.0M
			Bytes in policy "default-placement": 1.1P
			Objects in policy "default-placement-bytes": 0
			Bytes in policy "default-placement-bytes": 0
			Meta Quota-Containers: 1000
			X-Timestamp: 1745522890.88092
			X-Account-Bytes-Used-Actual: 1287786000326656
			X-Trans-Id: tx0000058e846920f427dfe-00680a90ca-83214639-default
			X-Openstack-Request-Id: tx0000058e846920f427dfe-00680a90ca-83214639-default
			Accept-Ranges: bytes
			Content-Type: text/plain; charset=utf-8
			Server: Ceph Object Gateway (squid)
			Connection: close

			</syntaxhighlight>


			You see here an account with 11 Containers (Swift's Containers are S3 Buckets) 2 Million objects, and 1.1 PB used out of its 1.5 PB quota.

			=== The S3 Api ===

			==What an Object Store is and isn't==
	(from https://github.com/s3fs-fuse/s3fs-fuse?tab=readme-ov-file#limitations)		(from https://github.com/s3fs-fuse/s3fs-fuse?tab=readme-ov-file#limitations)

	Generally, an Object Store cannot offer the same performance or semantics as a local file system. More specifically:		Generally, an Object Store cannot offer the same performance or semantics as a local file system. More specifically:

	* random writes or appends to files require rewriting the entire object, optimized with multi-part upload copy		*random writes or appends to files require rewriting the entire object, optimized with multi-part upload copy
	* metadata operations such as listing directories have poor performance due to network latency		*metadata operations such as listing directories have poor performance due to network latency
	* non-AWS providers may have eventual consistency so reads can temporarily yield stale data (AWS offers read-after-write consistency since Dec 2020)		*non-AWS providers may have eventual consistency so reads can temporarily yield stale data (AWS offers read-after-write consistency since Dec 2020)
	* no atomic renames of files or directories		*no atomic renames of files or directories
	* no coordination between multiple clients mounting the same bucket		*no coordination between multiple clients mounting the same bucket
	* no hard links		*no hard links
	* inotify detects only local modifications, not external ones by other clients or tools		*inotify detects only local modifications, not external ones by other clients or tools