Rclone - Revision history

Dbrownlee at 19:15, 6 March 2026

2026-03-06T19:15:33Z

← Older revision		Revision as of 19:15, 6 March 2026
Line 43:		Line 43:
	# For example:		# For example:
	#rclone mount c3g-data-repos:ihec_data /mnt/ihec_data_objstr --daemon --daemon-wait 0 --allow-other --read-only		#rclone mount c3g-data-repos:ihec_data /mnt/ihec_data_objstr --daemon --daemon-wait 0 --allow-other --read-only
	</syntaxhighlight>A service may be used to auto-mount the Object Store on boot with a service file (in /etc/systemd/system/).<syntaxhighlight lang="bash">		</syntaxhighlight>Unmount with:

			<code>fusermount -u /path/to/local/mount</code>


			A service may be used to auto-mount the Object Store on boot with a service file (in /etc/systemd/system/).<syntaxhighlight lang="bash">
	# Mount the ihec_data_objstr, even after a restart		# Mount the ihec_data_objstr, even after a restart
	[Unit]		[Unit]

Poq: /* I cannot upload a file larger than 48GB. */

2026-03-01T14:11:09Z

I cannot upload a file larger than 48GB.

← Older revision		Revision as of 14:11, 1 March 2026
Line 96:		Line 96:
	Another way is to lower the maximum number of parts in a multipart upload using [https://rclone.org/s3/#s3-max-upload-parts --s3-max-upload-parts], for example: <code>--s3-max-upload-parts 1000</code>.		Another way is to lower the maximum number of parts in a multipart upload using [https://rclone.org/s3/#s3-max-upload-parts --s3-max-upload-parts], for example: <code>--s3-max-upload-parts 1000</code>.

	Note that you need the ram of your computer to be larger ~~that~~ chunks.		Note that you need the ram of your computer to be larger than chunks.

Poq: /* I cannot upload a file larger than 48GB. */

2026-03-01T14:10:30Z

I cannot upload a file larger than 48GB.

← Older revision		Revision as of 14:10, 1 March 2026
Line 86:		Line 86:

	=== I cannot upload a file larger than 48GB. ===		=== I cannot upload a file larger than 48GB. ===
	In some situations, rclone is not able to guess the size of the file to upload and use the default value of <code>--s3-chunk-size 5M</code> to ~~spit~~ and upload the file to the bucket. But since the server has a 10,000 chunk limit, the upload crashes.		In some situations, rclone is not able to guess the size of the file to upload and use the default value of <code>--s3-chunk-size 5M</code> to split and upload the file to the bucket. But since the server has a 10,000 chunk limit, the upload crashes.

	You can solve that by setting a larger value:		You can solve that by setting a larger value:

Sbonami at 23:32, 11 November 2025

2025-11-11T23:32:56Z

← Older revision		Revision as of 23:32, 11 November 2025
Line 85:		Line 85:
	== No problems, only solutions ==		== No problems, only solutions ==

	1. I cannot upload file larger than 48GB.		=== I cannot upload a file larger than 48GB. ===
	In some ~~situation~~ rclone is not able to guess the size of the file to upload and use the default value of`--s3-chunk-size 5M` to spit and upload file to the bucket. But since the server has a 10,000 chunk limit, the upload crashes. You can solve that by setting a larger value:		In some situations, rclone is not able to guess the size of the file to upload and use the default value of <code>--s3-chunk-size 5M</code> to spit and upload the file to the bucket. But since the server has a 10,000 chunk limit, the upload crashes.

			You can solve that by setting a larger value:

	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
	$rclone copy --s3-chunk-size 50M my-large-file.cram my-project:test		rclone copy --s3-chunk-size 50M my-large-file.cram my-project:test
	</syntaxhighlight>		</syntaxhighlight>

			Another way is to lower the maximum number of parts in a multipart upload using [https://rclone.org/s3/#s3-max-upload-parts --s3-max-upload-parts], for example: <code>--s3-max-upload-parts 1000</code>.

	Note that you need the ram of your computer to be larger that chunks.		Note that you need the ram of your computer to be larger that chunks.

Dbrownlee at 17:49, 24 April 2025

2025-04-24T17:49:42Z

← Older revision		Revision as of 17:49, 24 April 2025
Line 63:		Line 63:
	[Install]		[Install]
	WantedBy=multi-user.target		WantedBy=multi-user.target
	</syntaxhighlight>		</syntaxhighlight>enable and start the service.

	== Mounting a public Object Store without using credentials ==		== Mounting a public Object Store without using credentials ==

Dbrownlee at 17:46, 24 April 2025

2025-04-24T17:46:56Z

← Older revision		Revision as of 17:46, 24 April 2025
Line 66:		Line 66:

	== Mounting a public Object Store without using credentials ==		== Mounting a public Object Store without using credentials ==
	Public Object Stores may be accessed or mounted as read-only without the use of Open Stack credentials. This relies on a bucket syntax ~~that includes~~ the Open Stack project ID.		Public Object Stores may be accessed or mounted as read-only without the use of Open Stack credentials. This relies on a bucket syntax prepended with the Open Stack project ID.

	Your ~/.config/rclone/rclone.conf need not contain an access_key_id and secret_access_key but only:<syntaxhighlight lang="ini" line="1">		Your ~/.config/rclone/rclone.conf need not contain an access_key_id and secret_access_key but only:<syntaxhighlight lang="ini" line="1">

Dbrownlee: Added mount (public and private) sections.

2025-04-24T17:45:56Z

Added mount (public and private) sections.

@@ Line 1: / Line 1: @@
-[https://rclone.org/ Rclone] is a powerful client that can interact with multiple storage backends, it offers a good support for our Ceph version of the S3 api and has good speed transfer out of the box.
+[https://rclone.org/ Rclone] is a powerful client that can interact with multiple storage backends, it offers a good support for our Ceph version of the S3 api and has good speed transfer out of the box.  It can also be used to mount an Object Store as traditional block file storage.
 == Configuration ==
-The first thing to do after [https://rclone.org/downloads/ having installed rclone] is to [[Store_and_Share_data#Configuring_S3_access|get your S3 <code>id key</code> and <code>secret</code>]]
+First [https://rclone.org/downloads/ download rclone] or use the [https://rclone.org/install/#script-installation script installation].  Then [[Store_and_Share_data#Configuring_S3_access|get your S3 <code>id key</code> and <code>secret</code>]] from Open Stack.
-You can then create the following file:
+Create the following file:
    ~/.config/rclone/rclone.conf
@@ Line 33: / Line 33: @@
 $rclone ls  my-project:test/
 my-file.txt
 </syntaxhighlight>

Poq: ^

2025-04-15T22:51:00Z

← Older revision		Revision as of 22:51, 15 April 2025
Line 3:		Line 3:
	== Configuration ==		== Configuration ==

	The first thing to do is to [[Store_and_Share_data#Configuring_S3_access\|get your S3 <code>id key</code> and <code>secret</code>]]		The first thing to do after [https://rclone.org/downloads/ having installed rclone] is to [[Store_and_Share_data#Configuring_S3_access\|get your S3 <code>id key</code> and <code>secret</code>]]

			You can then create the following file:

			~/.config/rclone/rclone.conf
			<syntaxhighlight lang="ini" line="1">
			[my-project]
			type = s3
			provider = Other
			env_auth = false
			access_key_id = <S3 ID from previous step>
			secret_access_key = <S3 secret from previous step>
			endpoint = https://objets.juno.calculquebec.ca
			acl = private
			</syntaxhighlight>

			You can then list current bucket, create a bucket and then copy a file into it,

			<syntaxhighlight lang="bash">
			$rclone lsd my-project:
			-1 2024-01-19 14:12:34 -1 backups
			-1 2024-03-07 14:23:26 -1 my-bucket
			$rclone mkdir c3g-prod:test
			$rclone lsd my-project:
			-1 2024-01-19 14:12:34 -1 backups
			-1 2024-03-07 14:23:26 -1 my-bucket
			-1 2025-04-15 18:08:32 -1 test
			$rclone copy my-file.txt my-project:test
			$rclone ls my-project:test/
			12408 my-file.txt
			</syntaxhighlight>

			== No problems, only solutions ==

			1. I cannot upload file larger than 48GB.
			In some situation rclone is not able to guess the size of the file to upload and use the default value of`--s3-chunk-size 5M` to spit and upload file to the bucket. But since the server has a 10,000 chunk limit, the upload crashes. You can solve that by setting a larger value:

			<syntaxhighlight lang="bash">
			$rclone copy --s3-chunk-size 50M my-large-file.cram my-project:test
			</syntaxhighlight>

			Note that you need the ram of your computer to be larger that chunks.

Poq: Created page with "[https://rclone.org/ Rclone] is a powerful client that can interact with multiple storage backends, it offers a good support for our Ceph version of the S3 api and has good speed transfer out of the box. == Configuration == The first thing to do is to get your S3 `id key` and `secret`"

2025-04-15T21:17:26Z

Created page with "[https://rclone.org/ Rclone] is a powerful client that can interact with multiple storage backends, it offers a good support for our Ceph version of the S3 api and has good speed transfer out of the box. == Configuration == The first thing to do is to get your S3 <code>id key</code> and <code>secret</code>"

New page

[https://rclone.org/ Rclone] is a powerful client that can interact with multiple storage backends, it offers a good support for our Ceph version of the S3 api and has good speed transfer out of the box.

== Configuration ==

The first thing to do is to [[Store_and_Share_data#Configuring_S3_access|get your S3 <code>id key</code> and <code>secret</code>]]