12
edits
Rclone: Difference between revisions
Added mount (public and private) sections.
(^) |
(Added mount (public and private) sections.) |
||
| Line 1: | Line 1: | ||
[https://rclone.org/ Rclone] is a powerful client that can interact with multiple storage backends, it offers a good support for our Ceph version of the S3 api and has good speed transfer out of the box. | [https://rclone.org/ Rclone] is a powerful client that can interact with multiple storage backends, it offers a good support for our Ceph version of the S3 api and has good speed transfer out of the box. It can also be used to mount an Object Store as traditional block file storage. | ||
== Configuration == | == Configuration == | ||
First [https://rclone.org/downloads/ download rclone] or use the [https://rclone.org/install/#script-installation script installation]. Then [[Store_and_Share_data#Configuring_S3_access|get your S3 <code>id key</code> and <code>secret</code>]] from Open Stack. | |||
Create the following file: | |||
~/.config/rclone/rclone.conf | ~/.config/rclone/rclone.conf | ||
| Line 33: | Line 33: | ||
$rclone ls my-project:test/ | $rclone ls my-project:test/ | ||
12408 my-file.txt | 12408 my-file.txt | ||
</syntaxhighlight> | |||
== Mounting an Object Store == | |||
To allow mounting by non-root users, in /etc/fuse.conf, uncomment:<syntaxhighlight lang="bash"> | |||
user_allow_other | |||
</syntaxhighlight>Mount the Object Store in daemon mode with:<syntaxhighlight lang="bash"> | |||
rclone mount <rclone config block>:<bucket> /path/to/mount/dir --daemon --daemon-wait 0 --allow-other --read-only | |||
# For example: | |||
#rclone mount c3g-data-repos:ihec_data /mnt/ihec_data_objstr --daemon --daemon-wait 0 --allow-other --read-only | |||
</syntaxhighlight>A service may be used to auto-mount the Object Store on boot with a service file (in /etc/systemd/system/).<syntaxhighlight lang="bash"> | |||
# Mount the ihec_data_objstr, even after a restart | |||
[Unit] | |||
Description=My Object Store automount | |||
After=network.target | |||
[Service] | |||
ExecStart=/usr/bin/rclone mount <rclone config block>:<bucket> /path/to/mount/point/dir --no-modtime --fast-list --transfers 50 --checkers 50 --allow-other --read-only | |||
# For example: | |||
# ExecStart=/usr/bin/rclone mount c3g-data-repos:ihec_data /mnt/ihec_data_objstr --no-modtime --fast-list --transfers 50 --checkers 50 --allow-other --read-only | |||
ExecStop=/usr/bin/fusermount -u /mnt/ihec_data_objstr | |||
Restart=always | |||
SyslogIdentifier=ihec_data_objstr | |||
User=ihec | |||
Group=ihec | |||
Environment=RCLONE_CONFIG=/home/ihec/.config/rclone/rclone.conf | |||
TimeoutStopSec=30 | |||
[Install] | |||
WantedBy=multi-user.target | |||
</syntaxhighlight> | |||
== Mounting a public Object Store without using credentials == | |||
Public Object Stores may be accessed or mounted as read-only without the use of Open Stack credentials. This relies on a bucket syntax that includes the Open Stack project ID. | |||
Your ~/.config/rclone/rclone.conf need not contain an access_key_id and secret_access_key but only:<syntaxhighlight lang="ini" line="1"> | |||
[my-public-project] | |||
type = s3 | |||
provider = Other | |||
env_auth = false | |||
endpoint = https://objets.juno.calculquebec.ca | |||
</syntaxhighlight>Then combine the OS project ID and the bucket name like so:<syntaxhighlight lang="bash"> | |||
rclone lsd my-public-project:<OS project ID>:<bucket name> | |||
# For example: | |||
# rclone lsd my-public-project:d5f8b8e8e3e2442f81573b2f0951013b:ihec_data | |||
# or | |||
# rclone mount my-public-project:d5f8b8e8e3e2442f81573b2f0951013b:ihec_data /mnt/ihec_data_objstr --daemon --daemon-wait 0 --allow-other --read-only | |||
</syntaxhighlight> | </syntaxhighlight> | ||