Rclone

Rclone is a powerful client that can interact with multiple storage backends, it offers a good support for our Ceph version of the S3 api and has good speed transfer out of the box.

Configuration

The first thing to do after having installed rclone is to get your S3 id key and secret

You can then create the following file:

 ~/.config/rclone/rclone.conf

[my-project]
type = s3
provider = Other
env_auth = false
access_key_id = <S3 ID from previous step>
secret_access_key = <S3 secret from previous step>
endpoint = https://objets.juno.calculquebec.ca
acl = private

You can then list current bucket, create a bucket and then copy a file into it,

$rclone lsd my-project:
          -1 2024-01-19 14:12:34        -1 backups
          -1 2024-03-07 14:23:26        -1 my-bucket
$rclone mkdir   c3g-prod:test
$rclone lsd my-project:
          -1 2024-01-19 14:12:34        -1 backups
          -1 2024-03-07 14:23:26        -1 my-bucket
          -1 2025-04-15 18:08:32        -1 test
$rclone copy my-file.txt my-project:test
$rclone ls  my-project:test/
    12408 my-file.txt

No problems, only solutions

1. I cannot upload file larger than 48GB.

In some situation rclone is not able to guess the size of the file to upload and use the default value of`--s3-chunk-size 5M` to spit and upload file to the bucket. But since the server has a 10,000 chunk limit, the upload crashes. You can solve that by setting a larger  value:

$rclone copy --s3-chunk-size 50M my-large-file.cram  my-project:test

Note that you need the ram of your computer to be larger that chunks.