123
edits
Rclone: Difference between revisions
^
(Created page with "[https://rclone.org/ Rclone] is a powerful client that can interact with multiple storage backends, it offers a good support for our Ceph version of the S3 api and has good speed transfer out of the box. == Configuration == The first thing to do is to get your S3 <code>id key</code> and <code>secret</code>") |
(^) |
||
| Line 3: | Line 3: | ||
== Configuration == | == Configuration == | ||
The first thing to do is to [[Store_and_Share_data#Configuring_S3_access|get your S3 <code>id key</code> and <code>secret</code>]] | The first thing to do after [https://rclone.org/downloads/ having installed rclone] is to [[Store_and_Share_data#Configuring_S3_access|get your S3 <code>id key</code> and <code>secret</code>]] | ||
You can then create the following file: | |||
~/.config/rclone/rclone.conf | |||
<syntaxhighlight lang="ini" line="1"> | |||
[my-project] | |||
type = s3 | |||
provider = Other | |||
env_auth = false | |||
access_key_id = <S3 ID from previous step> | |||
secret_access_key = <S3 secret from previous step> | |||
endpoint = https://objets.juno.calculquebec.ca | |||
acl = private | |||
</syntaxhighlight> | |||
You can then list current bucket, create a bucket and then copy a file into it, | |||
<syntaxhighlight lang="bash"> | |||
$rclone lsd my-project: | |||
-1 2024-01-19 14:12:34 -1 backups | |||
-1 2024-03-07 14:23:26 -1 my-bucket | |||
$rclone mkdir c3g-prod:test | |||
$rclone lsd my-project: | |||
-1 2024-01-19 14:12:34 -1 backups | |||
-1 2024-03-07 14:23:26 -1 my-bucket | |||
-1 2025-04-15 18:08:32 -1 test | |||
$rclone copy my-file.txt my-project:test | |||
$rclone ls my-project:test/ | |||
12408 my-file.txt | |||
</syntaxhighlight> | |||
== No problems, only solutions == | |||
1. I cannot upload file larger than 48GB. | |||
In some situation rclone is not able to guess the size of the file to upload and use the default value of`--s3-chunk-size 5M` to spit and upload file to the bucket. But since the server has a 10,000 chunk limit, the upload crashes. You can solve that by setting a larger value: | |||
<syntaxhighlight lang="bash"> | |||
$rclone copy --s3-chunk-size 50M my-large-file.cram my-project:test | |||
</syntaxhighlight> | |||
Note that you need the ram of your computer to be larger that chunks. | |||