SD4H wiki - User contributions [en]

https://wiki.c3g-app.sd4h.ca/api.php?action=feedcontributions&feedformat=atom&user=Dbrownlee SD4H wiki - User contributions [en] 2026-04-24T10:16:49Z User contributions MediaWiki 1.39.17 https://wiki.c3g-app.sd4h.ca/index.php?title=Rclone&diff=221 Rclone 2026-03-06T19:15:33Z

<p>Dbrownlee: </p> <hr /> <div>[https://rclone.org/ Rclone] is a powerful client that can interact with multiple storage backends, it offers a good support for our Ceph version of the S3 api and has good speed transfer out of the box. It can also be used to mount an Object Store as traditional block file storage. <br /> <br /> == Configuration ==<br /> <br /> First [https://rclone.org/downloads/ download rclone] or use the [https://rclone.org/install/#script-installation script installation]. Then [[Store_and_Share_data#Configuring_S3_access|get your S3 <code>id key</code> and <code>secret</code>]] from Open Stack.<br /> <br /> Create the following file: <br /> <br /> ~/.config/rclone/rclone.conf<br /> <syntaxhighlight lang="ini" line="1"><br /> [my-project]<br /> type = s3<br /> provider = Other<br /> env_auth = false<br /> access_key_id = <S3 ID from previous step><br /> secret_access_key = <S3 secret from previous step><br /> endpoint = https://objets.juno.calculquebec.ca<br /> acl = private<br /> </syntaxhighlight><br /> <br /> You can then list current bucket, create a bucket and then copy a file into it,<br /> <br /> <syntaxhighlight lang="bash"><br /> $rclone lsd my-project:<br /> -1 2024-01-19 14:12:34 -1 backups<br /> -1 2024-03-07 14:23:26 -1 my-bucket<br /> $rclone mkdir c3g-prod:test<br /> $rclone lsd my-project:<br /> -1 2024-01-19 14:12:34 -1 backups<br /> -1 2024-03-07 14:23:26 -1 my-bucket<br /> -1 2025-04-15 18:08:32 -1 test<br /> $rclone copy my-file.txt my-project:test<br /> $rclone ls my-project:test/<br /> 12408 my-file.txt<br /> </syntaxhighlight><br /> <br /> == Mounting an Object Store ==<br /> To allow mounting by non-root users, in /etc/fuse.conf, uncomment:<syntaxhighlight lang="bash"><br /> user_allow_other<br /> <br /> </syntaxhighlight>Mount the Object Store in daemon mode with:<syntaxhighlight lang="bash"><br /> rclone mount <rclone config block>:<bucket> /path/to/mount/dir --daemon --daemon-wait 0 --allow-other --read-only<br /> # For example:<br /> #rclone mount c3g-data-repos:ihec_data /mnt/ihec_data_objstr --daemon --daemon-wait 0 --allow-other --read-only<br /> </syntaxhighlight>Unmount with:<br /> <br /> <code>fusermount -u /path/to/local/mount</code><br /> <br /> <br /> A service may be used to auto-mount the Object Store on boot with a service file (in /etc/systemd/system/).<syntaxhighlight lang="bash"><br /> # Mount the ihec_data_objstr, even after a restart<br /> [Unit]<br /> Description=My Object Store automount<br /> After=network.target<br /> <br /> [Service]<br /> ExecStart=/usr/bin/rclone mount <rclone config block>:<bucket> /path/to/mount/point/dir --no-modtime --fast-list --transfers 50 --checkers 50 --allow-other --read-only<br /> # For example:<br /> # ExecStart=/usr/bin/rclone mount c3g-data-repos:ihec_data /mnt/ihec_data_objstr --no-modtime --fast-list --transfers 50 --checkers 50 --allow-other --read-only<br /> ExecStop=/usr/bin/fusermount -u /mnt/ihec_data_objstr<br /> Restart=always<br /> SyslogIdentifier=ihec_data_objstr<br /> User=ihec<br /> Group=ihec<br /> Environment=RCLONE_CONFIG=/home/ihec/.config/rclone/rclone.conf<br /> TimeoutStopSec=30<br /> <br /> [Install]<br /> WantedBy=multi-user.target<br /> </syntaxhighlight>enable and start the service.<br /> <br /> == Mounting a public Object Store without using credentials ==<br /> Public Object Stores may be accessed or mounted as read-only without the use of Open Stack credentials. This relies on a bucket syntax prepended with the Open Stack project ID.<br /> <br /> Your ~/.config/rclone/rclone.conf need not contain an access_key_id and secret_access_key but only:<syntaxhighlight lang="ini" line="1"><br /> [my-public-project]<br /> type = s3<br /> provider = Other<br /> env_auth = false<br /> endpoint = https://objets.juno.calculquebec.ca<br /> <br /> </syntaxhighlight>Then combine the OS project ID and the bucket name like so:<syntaxhighlight lang="bash"><br /> rclone lsd my-public-project:<OS project ID>:<bucket name><br /> # For example:<br /> # rclone lsd my-public-project:d5f8b8e8e3e2442f81573b2f0951013b:ihec_data<br /> # or<br /> # rclone mount my-public-project:d5f8b8e8e3e2442f81573b2f0951013b:ihec_data /mnt/ihec_data_objstr --daemon --daemon-wait 0 --allow-other --read-only<br /> </syntaxhighlight><br /> <br /> == No problems, only solutions ==<br /> <br /> === I cannot upload a file larger than 48GB. === <br /> In some situations, rclone is not able to guess the size of the file to upload and use the default value of <code>--s3-chunk-size 5M</code> to split and upload the file to the bucket. But since the server has a 10,000 chunk limit, the upload crashes.<br /> <br /> You can solve that by setting a larger value:<br /> <br /> <syntaxhighlight lang="bash"><br /> rclone copy --s3-chunk-size 50M my-large-file.cram my-project:test<br /> </syntaxhighlight><br /> <br /> Another way is to lower the maximum number of parts in a multipart upload using [https://rclone.org/s3/#s3-max-upload-parts --s3-max-upload-parts], for example: <code>--s3-max-upload-parts 1000</code>.<br /> <br /> Note that you need the ram of your computer to be larger than chunks.</div>

Dbrownlee https://wiki.c3g-app.sd4h.ca/index.php?title=Globus&diff=212 Globus 2025-12-15T17:38:58Z

<p>Dbrownlee: Browser compatibility issues.</p> <hr /> <div>== Requesting access ==<br /> We offer the possibility of managing you Object Storage space with Globus. This is not a default offering and the configuration needs to be activated by one of our platform administrator. You can request access at juno@calculquebec.ca.<br /> <br /> == Globus Web App Browser Compatibility Issues ==<br /> Despite Globus's claims to be compatible with most evergreen browsers, compatibility issues have been personally experienced with Firefox (146.0, 5-Dec-2025). Uploading and downloading through the Web App did not function. Chrome was used successfully as a work-around.<br /> <br /> == Get S3 API key ==<br /> <br /> The data access is a two level step. SD4H needs to give you access, you also need to generate you own key to complete the procedure. The procedure to generate [[Store_and_Share_data#Create_the_S3_(ec2)_Credentials|S3 credentials is documented on another page]]. The important point here is that the ''access'' and ''secret'' keys will are named ''AWS IAM Access Key ID'' and ''AWS IAM Secret Key'' respectively in the Globus interface.<br /> <br /> == Accessing My Data ==<br /> Once you have the confirmation that a Globus Collection has been configured on the SD4H side, you can connected to [https://app.globus.org the Globus app ] with your ''Digital Research Alliance of Canada'' account and look for the name of your OpenStack project in FILE MANAGER --> Collection search inbox. Here the project this example de project is named c3g-dev. <br /> <br /> [[File:Screenshot from 2025-02-10 15-54-48.png|caption|none|1200px|Search for you project's collection]]<br /> <br /> The collection will always be named <project name>_collection and be subscribed under SD4H Globus Automated Endpoint, look the the three pictograms to the left of the list, you will see the pad lock and the pantheon-like pictogram and the key, meaning respectively that the collection is in institutional High Availability mode and private.<br /> <br /> <br /> <br /> After accessing the collection you will see the following request: '''Your credential requires some initial setup'''. Click continue and enter there the ''AWS IAM Access Key ID'' and ''AWS IAM Secret Key'' you got from the the previous step.</div>

Dbrownlee https://wiki.c3g-app.sd4h.ca/index.php?title=Globus&diff=211 Globus 2025-12-15T17:34:29Z

<p>Dbrownlee: Minor syntax.</p> <hr /> <div>== Requesting access ==<br /> We offer the possibility of managing you Object Storage space with Globus. This is not a default offering and the configuration needs to be activated by one of our platform administrator. You can request access at juno@calculquebec.ca.<br /> <br /> == Get S3 API key ==<br /> <br /> The data access is a two level step. SD4H needs to give you access, you also need to generate you own key to complete the procedure. The procedure to generate [[Store_and_Share_data#Create_the_S3_(ec2)_Credentials|S3 credentials is documented on another page]]. The important point here is that the ''access'' and ''secret'' keys will are named ''AWS IAM Access Key ID'' and ''AWS IAM Secret Key'' respectively in the Globus interface.<br /> <br /> == Accessing My Data ==<br /> Once you have the confirmation that a Globus Collection has been configured on the SD4H side, you can connected to [https://app.globus.org the Globus app ] with your ''Digital Research Alliance of Canada'' account and look for the name of your OpenStack project in FILE MANAGER --> Collection search inbox. Here the project this example de project is named c3g-dev. <br /> <br /> [[File:Screenshot from 2025-02-10 15-54-48.png|caption|none|1200px|Search for you project's collection]]<br /> <br /> The collection will always be named <project name>_collection and be subscribed under SD4H Globus Automated Endpoint, look the the three pictograms to the left of the list, you will see the pad lock and the pantheon-like pictogram and the key, meaning respectively that the collection is in institutional High Availability mode and private.<br /> <br /> <br /> <br /> After accessing the collection you will see the following request: '''Your credential requires some initial setup'''. Click continue and enter there the ''AWS IAM Access Key ID'' and ''AWS IAM Secret Key'' you got from the the previous step.</div>

Dbrownlee https://wiki.c3g-app.sd4h.ca/index.php?title=SSH_to_a_server_via_Bastion&diff=188 SSH to a server via Bastion 2025-08-13T14:15:45Z

<p>Dbrownlee: /* Connecting to a server */</p> <hr /> <div>== Bastion in a nutshell ==<br /> [https://ovh.github.io/the-bastion/index.html The Bastion] is a single point of entry for SSH connections. Like an http reverse proxy for ssh, it is used to connect to other servers.<br /> <br /> Some servers in SD4H may not be publicly accessible from the public internet for security reasons.<br /> <br /> Bastion is used as an SSH proxy to connect users to servers.<br /> <br /> == User guide ==<br /> <br /> === Prerequisites ===<br /> To use Bastion, an administrator must create an account for you.<br /> <br /> Please include the public SSH key you will use to connect to Bastion in your request for an account.<br /> <br /> After creating your account, the administrator will get back to you with your Bastion user name and a proposed alias.<br /> <br /> === Creating the Bastion alias ===<br /> For your convenience, it is recommended to create an alias for the Bastion connection:<syntaxhighlight lang="bash"><br /> alias bssh='ssh -t <YOUR BASTION USERNAME>@bastion.sd4h.ca --'<br /> </syntaxhighlight>After adding the alias, you can connect to (or through) Bastion!<br /> <br /> === Connection types ===<br /> There are three types of connections you can make.<br /> <br /> # Interactive mode<br /> # Non-interactive mode<br /> # Directly to an external, registered server.<br /> In a direct connection, The Bastion acts as a transparent pass-through. This is the most convenient, likely and common way you will connect. See the "Connecting to a server" section below.<br /> <br /> <syntaxhighlight lang="bash"><br /> # Interactive mode (signs out when idle)<br /> bssh # SSH into Bastion<br /> help # runs the Bastion 'help' command<br /> <br /> # Non-interactive mode equivalent with '--osh'<br /> bssh --osh help<br /> </syntaxhighlight><br /> <br /> === Permissions ===<br /> Bastion uses the principle of least privileges, meaning that users must be granted explicit access to servers.<br /> <br /> As such, a fresh Bastion user will not have access to anything at first, accesses must be granted by an administrator.<br /> <br /> === Listing your server accesses ===<br /> The <code>selfListAccesses</code> command lists the servers you have access to in Bastion.<br /> <br /> To view your accesses, simply run the command:<syntaxhighlight lang="bash"><br /> bssh --osh selfListAccesses<br /> <br /> # Dear <USERNAME>, you have access to the following servers:<br /> # IP PORT USER ACCESS-BY ADDED-BY ADDED-AT<br /> # -------------- ---- -------------- ---------------------- ---------- ----------<br /> # <SERVER 1 IP> 22 <SERVER 1 USER> <GROUP>(group-member) vrocheleau 2024-09-11<br /> # <SERVER 2 IP> 22 <SERVER 2 USER> personal vrocheleau 2024-09-11<br /> <br /> </syntaxhighlight><br /> <br /> === Connecting to a server ===<br /> You can connect directly to any server that is listed by the <code>selfListAccesses</code> command.<br /> <br /> Taking the example output from the previous section:<syntaxhighlight lang="bash"><br /> # connect to "SERVER 1"<br /> bssh <SERVER 1 USER>@<SERVER 1 IP><br /> <br /> # connect to "SERVER 2"<br /> bssh <SERVER 2 USER>@<SERVER 2 IP><br /> </syntaxhighlight>OpenStack provides name resolution for VMs on the same network. If the Bastion server is on the same OpenStack network as the destination servers, you may use the VM name rather than its IP address. This makes using bssh essentially identical to using ssh (though rather than a destination domain name, the VM name is used). <syntaxhighlight lang="bash"><br /> bssh <SERVER 1 USER>@<SERVER 1 HOSTNAME><br /> <br /> </syntaxhighlight><br /> <br /> === Typical Commands ===<br /> The Bastion has a lot of commands available. Autocomplete is very helpful and removes the need to remember a whole new set of CLI commands. For a typical user, the [potentially] most relevant are: <syntaxhighlight lang="bash"><br /> # Show a help message, including available commands.<br /> help<br /> # Basic info about your account.<br /> info<br /> # List accessible servers.<br /> selfListAccess<br /> # Generate scp passthrough script.<br /> scp<br /> </syntaxhighlight><br /> <br /> === Using scp, sftp, rsync through The Bastion ===<br /> It is possible to scp files in both directions through The Bastion using a special script that The Bastion will generate for you. Follow the [https://ovh.github.io/the-bastion/plugins/open/scp.html scp setup directions] found in The Bastion documentation.<br /> <br /> Check the scp-via-bastion script The Bastion generates - it may need some minor tweaking.<syntaxhighlight lang="bash"><br /> # Check<br /> BASTION_CMD="ssh davidbr@bastion-candig "<br /> # Change to FQDN or IP address, if needed.<br /> BASTION_CMD="ssh davidbr@198.168.188.147 " # Or @<DOMAIN NAME><br /> </syntaxhighlight>The scp-via-bastion script may also be renamed (or aliased) and moved from $HOME to a directory within the user's $PATH (such as ~/.local/bin/).<br /> <br /> === Access types ===<br /> The Bastion provides both Group and Personal access methodologies. Which method you use depends on The Bastion configurations. Your connections will work the same no matter which method is used, so from a user perspective, the distinction is moot.</div>

Dbrownlee https://wiki.c3g-app.sd4h.ca/index.php?title=SSH_to_a_server_via_Bastion&diff=187 SSH to a server via Bastion 2025-08-11T15:52:54Z

<p>Dbrownlee: /* Typical Commands */</p> <hr /> <div>== Bastion in a nutshell ==<br /> [https://ovh.github.io/the-bastion/index.html The Bastion] is a single point of entry for SSH connections. Like an http reverse proxy for ssh, it is used to connect to other servers.<br /> <br /> Some servers in SD4H may not be publicly accessible from the public internet for security reasons.<br /> <br /> Bastion is used as an SSH proxy to connect users to servers.<br /> <br /> == User guide ==<br /> <br /> === Prerequisites ===<br /> To use Bastion, an administrator must create an account for you.<br /> <br /> Please include the public SSH key you will use to connect to Bastion in your request for an account.<br /> <br /> After creating your account, the administrator will get back to you with your Bastion user name and a proposed alias.<br /> <br /> === Creating the Bastion alias ===<br /> For your convenience, it is recommended to create an alias for the Bastion connection:<syntaxhighlight lang="bash"><br /> alias bssh='ssh -t <YOUR BASTION USERNAME>@bastion.sd4h.ca --'<br /> </syntaxhighlight>After adding the alias, you can connect to (or through) Bastion!<br /> <br /> === Connection types ===<br /> There are three types of connections you can make.<br /> <br /> # Interactive mode<br /> # Non-interactive mode<br /> # Directly to an external, registered server.<br /> In a direct connection, The Bastion acts as a transparent pass-through. This is the most convenient, likely and common way you will connect. See the "Connecting to a server" section below.<br /> <br /> <syntaxhighlight lang="bash"><br /> # Interactive mode (signs out when idle)<br /> bssh # SSH into Bastion<br /> help # runs the Bastion 'help' command<br /> <br /> # Non-interactive mode equivalent with '--osh'<br /> bssh --osh help<br /> </syntaxhighlight><br /> <br /> === Permissions ===<br /> Bastion uses the principle of least privileges, meaning that users must be granted explicit access to servers.<br /> <br /> As such, a fresh Bastion user will not have access to anything at first, accesses must be granted by an administrator.<br /> <br /> === Listing your server accesses ===<br /> The <code>selfListAccesses</code> command lists the servers you have access to in Bastion.<br /> <br /> To view your accesses, simply run the command:<syntaxhighlight lang="bash"><br /> bssh --osh selfListAccesses<br /> <br /> # Dear <USERNAME>, you have access to the following servers:<br /> # IP PORT USER ACCESS-BY ADDED-BY ADDED-AT<br /> # -------------- ---- -------------- ---------------------- ---------- ----------<br /> # <SERVER 1 IP> 22 <SERVER 1 USER> <GROUP>(group-member) vrocheleau 2024-09-11<br /> # <SERVER 2 IP> 22 <SERVER 2 USER> personal vrocheleau 2024-09-11<br /> <br /> </syntaxhighlight><br /> <br /> === Connecting to a server ===<br /> You can connect directly to any server that is listed by the <code>selfListAccesses</code> command.<br /> <br /> Taking the example output from the previous section:<syntaxhighlight lang="bash"><br /> # connect to "SERVER 1"<br /> bssh <SERVER 1 USER>@<SERVER 1 IP><br /> <br /> # connect to "SERVER 2"<br /> bssh <SERVER 2 USER>@<SERVER 2 IP><br /> </syntaxhighlight>OpenStack provides name resolution for VMs on the same network. If the Bastion server is on the same OpenStack network as the destination servers, you may use the VM name rather than its IP address. This makes using bssh essentially identical to using ssh (though rather than a destination domain name, the VM name is used). <syntaxhighlight lang="bash"><br /> bssh <SERVER 1 USER>@<SERVER 1 NAME><br /> <br /> </syntaxhighlight><br /> <br /> === Typical Commands ===<br /> The Bastion has a lot of commands available. Autocomplete is very helpful and removes the need to remember a whole new set of CLI commands. For a typical user, the [potentially] most relevant are: <syntaxhighlight lang="bash"><br /> # Show a help message, including available commands.<br /> help<br /> # Basic info about your account.<br /> info<br /> # List accessible servers.<br /> selfListAccess<br /> # Generate scp passthrough script.<br /> scp<br /> </syntaxhighlight><br /> <br /> === Using scp, sftp, rsync through The Bastion ===<br /> It is possible to scp files in both directions through The Bastion using a special script that The Bastion will generate for you. Follow the [https://ovh.github.io/the-bastion/plugins/open/scp.html scp setup directions] found in The Bastion documentation.<br /> <br /> Check the scp-via-bastion script The Bastion generates - it may need some minor tweaking.<syntaxhighlight lang="bash"><br /> # Check<br /> BASTION_CMD="ssh davidbr@bastion-candig "<br /> # Change to FQDN or IP address, if needed.<br /> BASTION_CMD="ssh davidbr@198.168.188.147 " # Or @<DOMAIN NAME><br /> </syntaxhighlight>The scp-via-bastion script may also be renamed (or aliased) and moved from $HOME to a directory within the user's $PATH (such as ~/.local/bin/).<br /> <br /> === Access types ===<br /> The Bastion provides both Group and Personal access methodologies. Which method you use depends on The Bastion configurations. Your connections will work the same no matter which method is used, so from a user perspective, the distinction is moot.</div>

Dbrownlee https://wiki.c3g-app.sd4h.ca/index.php?title=SSH_to_a_server_via_Bastion&diff=186 SSH to a server via Bastion 2025-08-11T14:55:30Z

<p>Dbrownlee: /* Connecting to a server */</p> <hr /> <div>== Bastion in a nutshell ==<br /> [https://ovh.github.io/the-bastion/index.html The Bastion] is a single point of entry for SSH connections. Like an http reverse proxy for ssh, it is used to connect to other servers.<br /> <br /> Some servers in SD4H may not be publicly accessible from the public internet for security reasons.<br /> <br /> Bastion is used as an SSH proxy to connect users to servers.<br /> <br /> == User guide ==<br /> <br /> === Prerequisites ===<br /> To use Bastion, an administrator must create an account for you.<br /> <br /> Please include the public SSH key you will use to connect to Bastion in your request for an account.<br /> <br /> After creating your account, the administrator will get back to you with your Bastion user name and a proposed alias.<br /> <br /> === Creating the Bastion alias ===<br /> For your convenience, it is recommended to create an alias for the Bastion connection:<syntaxhighlight lang="bash"><br /> alias bssh='ssh -t <YOUR BASTION USERNAME>@bastion.sd4h.ca --'<br /> </syntaxhighlight>After adding the alias, you can connect to (or through) Bastion!<br /> <br /> === Connection types ===<br /> There are three types of connections you can make.<br /> <br /> # Interactive mode<br /> # Non-interactive mode<br /> # Directly to an external, registered server.<br /> In a direct connection, The Bastion acts as a transparent pass-through. This is the most convenient, likely and common way you will connect. See the "Connecting to a server" section below.<br /> <br /> <syntaxhighlight lang="bash"><br /> # Interactive mode (signs out when idle)<br /> bssh # SSH into Bastion<br /> help # runs the Bastion 'help' command<br /> <br /> # Non-interactive mode equivalent with '--osh'<br /> bssh --osh help<br /> </syntaxhighlight><br /> <br /> === Permissions ===<br /> Bastion uses the principle of least privileges, meaning that users must be granted explicit access to servers.<br /> <br /> As such, a fresh Bastion user will not have access to anything at first, accesses must be granted by an administrator.<br /> <br /> === Listing your server accesses ===<br /> The <code>selfListAccesses</code> command lists the servers you have access to in Bastion.<br /> <br /> To view your accesses, simply run the command:<syntaxhighlight lang="bash"><br /> bssh --osh selfListAccesses<br /> <br /> # Dear <USERNAME>, you have access to the following servers:<br /> # IP PORT USER ACCESS-BY ADDED-BY ADDED-AT<br /> # -------------- ---- -------------- ---------------------- ---------- ----------<br /> # <SERVER 1 IP> 22 <SERVER 1 USER> <GROUP>(group-member) vrocheleau 2024-09-11<br /> # <SERVER 2 IP> 22 <SERVER 2 USER> personal vrocheleau 2024-09-11<br /> <br /> </syntaxhighlight><br /> <br /> === Connecting to a server ===<br /> You can connect directly to any server that is listed by the <code>selfListAccesses</code> command.<br /> <br /> Taking the example output from the previous section:<syntaxhighlight lang="bash"><br /> # connect to "SERVER 1"<br /> bssh <SERVER 1 USER>@<SERVER 1 IP><br /> <br /> # connect to "SERVER 2"<br /> bssh <SERVER 2 USER>@<SERVER 2 IP><br /> </syntaxhighlight>OpenStack provides name resolution for VMs on the same network. If the Bastion server is on the same OpenStack network as the destination servers, you may use the VM name rather than its IP address. This makes using bssh essentially identical to using ssh (though rather than a destination domain name, the VM name is used). <syntaxhighlight lang="bash"><br /> bssh <SERVER 1 USER>@<SERVER 1 NAME><br /> <br /> </syntaxhighlight><br /> <br /> === Typical Commands ===<br /> The Bastion has a lot of commands available. Autocomplete is very helpful and removes the need to remember a whole new set of CLI commands. For a typical user, the [potentially] most relevant are: <syntaxhighlight lang="bash"><br /> # Show a help message, including available commands.<br /> help<br /> # Basic info<br /> info<br /> # List accessible servers.<br /> selfListAccess<br /> # Generate scp passthrough script<br /> scp<br /> </syntaxhighlight><br /> <br /> === Using scp, sftp, rsync through The Bastion ===<br /> It is possible to scp files in both directions through The Bastion using a special script that The Bastion will generate for you. Follow the [https://ovh.github.io/the-bastion/plugins/open/scp.html scp setup directions] found in The Bastion documentation.<br /> <br /> Check the scp-via-bastion script The Bastion generates - it may need some minor tweaking.<syntaxhighlight lang="bash"><br /> # Check<br /> BASTION_CMD="ssh davidbr@bastion-candig "<br /> # Change to FQDN or IP address, if needed.<br /> BASTION_CMD="ssh davidbr@bastion.candig.sd4h.ca " # Or @<IP ADDRESS><br /> </syntaxhighlight>The scp-via-bastion script may also be renamed (or aliased) and moved from $HOME to a directory within the user's $PATH (such as ~/.local/bin/).<br /> <br /> === Access types ===<br /> The Bastion provides both Group and Personal access methodologies. Which method you use depends on The Bastion configurations. Your connections will work the same no matter which method is used, so from a user perspective, the distinction is moot.</div>

Dbrownlee https://wiki.c3g-app.sd4h.ca/index.php?title=SSH_to_a_server_via_Bastion&diff=185 SSH to a server via Bastion 2025-08-11T14:34:37Z

<p>Dbrownlee: /* Using scp, sftp, rsync through The Bastion */</p> <hr /> <div>== Bastion in a nutshell ==<br /> [https://ovh.github.io/the-bastion/index.html The Bastion] is a single point of entry for SSH connections. Like an http reverse proxy for ssh, it is used to connect to other servers.<br /> <br /> Some servers in SD4H may not be publicly accessible from the public internet for security reasons.<br /> <br /> Bastion is used as an SSH proxy to connect users to servers.<br /> <br /> == User guide ==<br /> <br /> === Prerequisites ===<br /> To use Bastion, an administrator must create an account for you.<br /> <br /> Please include the public SSH key you will use to connect to Bastion in your request for an account.<br /> <br /> After creating your account, the administrator will get back to you with your Bastion user name and a proposed alias.<br /> <br /> === Creating the Bastion alias ===<br /> For your convenience, it is recommended to create an alias for the Bastion connection:<syntaxhighlight lang="bash"><br /> alias bssh='ssh -t <YOUR BASTION USERNAME>@bastion.sd4h.ca --'<br /> </syntaxhighlight>After adding the alias, you can connect to (or through) Bastion!<br /> <br /> === Connection types ===<br /> There are three types of connections you can make.<br /> <br /> # Interactive mode<br /> # Non-interactive mode<br /> # Directly to an external, registered server.<br /> In a direct connection, The Bastion acts as a transparent pass-through. This is the most convenient, likely and common way you will connect. See the "Connecting to a server" section below.<br /> <br /> <syntaxhighlight lang="bash"><br /> # Interactive mode (signs out when idle)<br /> bssh # SSH into Bastion<br /> help # runs the Bastion 'help' command<br /> <br /> # Non-interactive mode equivalent with '--osh'<br /> bssh --osh help<br /> </syntaxhighlight><br /> <br /> === Permissions ===<br /> Bastion uses the principle of least privileges, meaning that users must be granted explicit access to servers.<br /> <br /> As such, a fresh Bastion user will not have access to anything at first, accesses must be granted by an administrator.<br /> <br /> === Listing your server accesses ===<br /> The <code>selfListAccesses</code> command lists the servers you have access to in Bastion.<br /> <br /> To view your accesses, simply run the command:<syntaxhighlight lang="bash"><br /> bssh --osh selfListAccesses<br /> <br /> # Dear <USERNAME>, you have access to the following servers:<br /> # IP PORT USER ACCESS-BY ADDED-BY ADDED-AT<br /> # -------------- ---- -------------- ---------------------- ---------- ----------<br /> # <SERVER 1 IP> 22 <SERVER 1 USER> <GROUP>(group-member) vrocheleau 2024-09-11<br /> # <SERVER 2 IP> 22 <SERVER 2 USER> personal vrocheleau 2024-09-11<br /> <br /> </syntaxhighlight><br /> <br /> === Connecting to a server ===<br /> You can connect directly to any server that is listed by the <code>selfListAccesses</code> command.<br /> <br /> Taking the example output from the previous section:<syntaxhighlight lang="bash"><br /> # connect to "SERVER 1"<br /> bssh <SERVER 1 USER>@<SERVER 1 IP><br /> <br /> # connect to "SERVER 2"<br /> bssh <SERVER 2 USER>@<SERVER 2 IP><br /> </syntaxhighlight>OpenStack provides name resolution for VMs on the same network. If the Bastion server is on the same OpenStack network as the destination servers, you may use the VM name rather than its IP address. This makes using bssh essentially identical to using ssh (though rather than a destination domain name, the VM name is used). <syntaxhighlight lang="bash"><br /> bssh <SERVER 1 USER>@<SERVER 1 NAME><br /> <br /> </syntaxhighlight><br /> <br /> === Using scp, sftp, rsync through The Bastion ===<br /> It is possible to scp files in both directions through The Bastion using a special script that The Bastion will generate for you. Follow the [https://ovh.github.io/the-bastion/plugins/open/scp.html scp setup directions] found in The Bastion documentation.<br /> <br /> Check the scp-via-bastion script The Bastion generates - it may need some minor tweaking.<syntaxhighlight lang="bash"><br /> # Check<br /> BASTION_CMD="ssh davidbr@bastion-candig "<br /> # Change to FQDN or IP address, if needed.<br /> BASTION_CMD="ssh davidbr@bastion.candig.sd4h.ca " # Or @<IP ADDRESS><br /> </syntaxhighlight>The scp-via-bastion script may also be renamed (or aliased) and moved from $HOME to a directory within the user's $PATH (such as ~/.local/bin/).<br /> <br /> === Access types ===<br /> The Bastion provides both Group and Personal access methodologies. Which method you use depends on The Bastion configurations. Your connections will work the same no matter which method is used, so from a user perspective, the distinction is moot.</div>

Dbrownlee https://wiki.c3g-app.sd4h.ca/index.php?title=SSH_to_a_server_via_Bastion&diff=184 SSH to a server via Bastion 2025-08-07T21:29:40Z

<p>Dbrownlee: /* Using scp, sftp, rsync through The Bastion */</p> <hr /> <div>== Bastion in a nutshell ==<br /> [https://ovh.github.io/the-bastion/index.html The Bastion] is a single point of entry for SSH connections. Like an http reverse proxy for ssh, it is used to connect to other servers.<br /> <br /> Some servers in SD4H may not be publicly accessible from the public internet for security reasons.<br /> <br /> Bastion is used as an SSH proxy to connect users to servers.<br /> <br /> == User guide ==<br /> <br /> === Prerequisites ===<br /> To use Bastion, an administrator must create an account for you.<br /> <br /> Please include the public SSH key you will use to connect to Bastion in your request for an account.<br /> <br /> After creating your account, the administrator will get back to you with your Bastion user name and a proposed alias.<br /> <br /> === Creating the Bastion alias ===<br /> For your convenience, it is recommended to create an alias for the Bastion connection:<syntaxhighlight lang="bash"><br /> alias bssh='ssh -t <YOUR BASTION USERNAME>@bastion.sd4h.ca --'<br /> </syntaxhighlight>After adding the alias, you can connect to (or through) Bastion!<br /> <br /> === Connection types ===<br /> There are three types of connections you can make.<br /> <br /> # Interactive mode<br /> # Non-interactive mode<br /> # Directly to an external, registered server.<br /> In a direct connection, The Bastion acts as a transparent pass-through. This is the most convenient, likely and common way you will connect. See the "Connecting to a server" section below.<br /> <br /> <syntaxhighlight lang="bash"><br /> # Interactive mode (signs out when idle)<br /> bssh # SSH into Bastion<br /> help # runs the Bastion 'help' command<br /> <br /> # Non-interactive mode equivalent with '--osh'<br /> bssh --osh help<br /> </syntaxhighlight><br /> <br /> === Permissions ===<br /> Bastion uses the principle of least privileges, meaning that users must be granted explicit access to servers.<br /> <br /> As such, a fresh Bastion user will not have access to anything at first, accesses must be granted by an administrator.<br /> <br /> === Listing your server accesses ===<br /> The <code>selfListAccesses</code> command lists the servers you have access to in Bastion.<br /> <br /> To view your accesses, simply run the command:<syntaxhighlight lang="bash"><br /> bssh --osh selfListAccesses<br /> <br /> # Dear <USERNAME>, you have access to the following servers:<br /> # IP PORT USER ACCESS-BY ADDED-BY ADDED-AT<br /> # -------------- ---- -------------- ---------------------- ---------- ----------<br /> # <SERVER 1 IP> 22 <SERVER 1 USER> <GROUP>(group-member) vrocheleau 2024-09-11<br /> # <SERVER 2 IP> 22 <SERVER 2 USER> personal vrocheleau 2024-09-11<br /> <br /> </syntaxhighlight><br /> <br /> === Connecting to a server ===<br /> You can connect directly to any server that is listed by the <code>selfListAccesses</code> command.<br /> <br /> Taking the example output from the previous section:<syntaxhighlight lang="bash"><br /> # connect to "SERVER 1"<br /> bssh <SERVER 1 USER>@<SERVER 1 IP><br /> <br /> # connect to "SERVER 2"<br /> bssh <SERVER 2 USER>@<SERVER 2 IP><br /> </syntaxhighlight>OpenStack provides name resolution for VMs on the same network. If the Bastion server is on the same OpenStack network as the destination servers, you may use the VM name rather than its IP address. This makes using bssh essentially identical to using ssh (though rather than a destination domain name, the VM name is used). <syntaxhighlight lang="bash"><br /> bssh <SERVER 1 USER>@<SERVER 1 NAME><br /> <br /> </syntaxhighlight><br /> <br /> === Using scp, sftp, rsync through The Bastion ===<br /> It is possible to scp files in both directions through The Bastion using a special script that The Bastion will generate for you. Follow the [https://ovh.github.io/the-bastion/plugins/open/scp.html scp setup directions] found in The Bastion documentation.<br /> <br /> Check the scp-via-bastion script The Bastion generates - it may need some minor tweaking.<syntaxhighlight lang="bash"><br /> # Check<br /> BASTION_CMD="ssh davidbr@bastion-candig "<br /> # Change to FQDN or IP address, if needed.<br /> BASTION_CMD="ssh davidbr@bastion.candig.sd4h.ca " # Or @<IP ADDRESS><br /> </syntaxhighlight>The scp-via-bastion script may also be renamed (or aliased) and moved from $HOME to a directory within the user's $PATH such as ~/.local/bin/.<br /> <br /> === Access types ===<br /> The Bastion provides both Group and Personal access methodologies. Which method you use depends on The Bastion configurations. Your connections will work the same no matter which method is used, so from a user perspective, the distinction is moot.</div>

Dbrownlee https://wiki.c3g-app.sd4h.ca/index.php?title=SSH_to_a_server_via_Bastion&diff=183 SSH to a server via Bastion 2025-08-07T19:36:26Z

<p>Dbrownlee: /* Connection types */</p> <hr /> <div>== Bastion in a nutshell ==<br /> [https://ovh.github.io/the-bastion/index.html The Bastion] is a single point of entry for SSH connections. Like an http reverse proxy for ssh, it is used to connect to other servers.<br /> <br /> Some servers in SD4H may not be publicly accessible from the public internet for security reasons.<br /> <br /> Bastion is used as an SSH proxy to connect users to servers.<br /> <br /> == User guide ==<br /> <br /> === Prerequisites ===<br /> To use Bastion, an administrator must create an account for you.<br /> <br /> Please include the public SSH key you will use to connect to Bastion in your request for an account.<br /> <br /> After creating your account, the administrator will get back to you with your Bastion user name and a proposed alias.<br /> <br /> === Creating the Bastion alias ===<br /> For your convenience, it is recommended to create an alias for the Bastion connection:<syntaxhighlight lang="bash"><br /> alias bssh='ssh -t <YOUR BASTION USERNAME>@bastion.sd4h.ca --'<br /> </syntaxhighlight>After adding the alias, you can connect to (or through) Bastion!<br /> <br /> === Connection types ===<br /> There are three types of connections you can make.<br /> <br /> # Interactive mode<br /> # Non-interactive mode<br /> # Directly to an external, registered server.<br /> In a direct connection, The Bastion acts as a transparent pass-through. This is the most convenient, likely and common way you will connect. See the "Connecting to a server" section below.<br /> <br /> <syntaxhighlight lang="bash"><br /> # Interactive mode (signs out when idle)<br /> bssh # SSH into Bastion<br /> help # runs the Bastion 'help' command<br /> <br /> # Non-interactive mode equivalent with '--osh'<br /> bssh --osh help<br /> </syntaxhighlight><br /> <br /> === Permissions ===<br /> Bastion uses the principle of least privileges, meaning that users must be granted explicit access to servers.<br /> <br /> As such, a fresh Bastion user will not have access to anything at first, accesses must be granted by an administrator.<br /> <br /> === Listing your server accesses ===<br /> The <code>selfListAccesses</code> command lists the servers you have access to in Bastion.<br /> <br /> To view your accesses, simply run the command:<syntaxhighlight lang="bash"><br /> bssh --osh selfListAccesses<br /> <br /> # Dear <USERNAME>, you have access to the following servers:<br /> # IP PORT USER ACCESS-BY ADDED-BY ADDED-AT<br /> # -------------- ---- -------------- ---------------------- ---------- ----------<br /> # <SERVER 1 IP> 22 <SERVER 1 USER> <GROUP>(group-member) vrocheleau 2024-09-11<br /> # <SERVER 2 IP> 22 <SERVER 2 USER> personal vrocheleau 2024-09-11<br /> <br /> </syntaxhighlight><br /> <br /> === Connecting to a server ===<br /> You can connect directly to any server that is listed by the <code>selfListAccesses</code> command.<br /> <br /> Taking the example output from the previous section:<syntaxhighlight lang="bash"><br /> # connect to "SERVER 1"<br /> bssh <SERVER 1 USER>@<SERVER 1 IP><br /> <br /> # connect to "SERVER 2"<br /> bssh <SERVER 2 USER>@<SERVER 2 IP><br /> </syntaxhighlight>OpenStack provides name resolution for VMs on the same network. If the Bastion server is on the same OpenStack network as the destination servers, you may use the VM name rather than its IP address. This makes using bssh essentially identical to using ssh (though rather than a destination domain name, the VM name is used). <syntaxhighlight lang="bash"><br /> bssh <SERVER 1 USER>@<SERVER 1 NAME><br /> <br /> </syntaxhighlight><br /> <br /> === Using scp, sftp, rsync through The Bastion ===<br /> It is possible to scp files in both directions through The Bastion using a special script that The Bastion will generate for you. Follow the [https://ovh.github.io/the-bastion/plugins/open/scp.html scp setup directions] found in The Bastion documentation.<br /> <br /> The scp-via-bastion script The Bastion generates needs some minor tweaking.<br /> From:<br /> BASTION_CMD="ssh davidbr@bastion-candig "<br /> To:<br /> BASTION_CMD="ssh davidbr@bastion.candig.sd4h.ca " # Or @<IP ADDRESS><br /> The scp-via-bastion script may also be renamed (or aliased) and moved to a directory present in the user's $PATH rather than $HOME.<br /> <br /> === Access types ===<br /> The Bastion provides both Group and Personal access methodologies. Which method you use depends on The Bastion configurations. Your connections will work the same no matter which method is used, so from a user perspective, the distinction is moot.</div>

Dbrownlee https://wiki.c3g-app.sd4h.ca/index.php?title=SSH_to_a_server_via_Bastion&diff=182 SSH to a server via Bastion 2025-08-07T18:48:10Z

<p>Dbrownlee: /* Bastion in a nutshell */</p> <hr /> <div>== Bastion in a nutshell ==<br /> [https://ovh.github.io/the-bastion/index.html The Bastion] is a single point of entry for SSH connections. Like an http reverse proxy for ssh, it is used to connect to other servers.<br /> <br /> Some servers in SD4H may not be publicly accessible from the public internet for security reasons.<br /> <br /> Bastion is used as an SSH proxy to connect users to servers.<br /> <br /> == User guide ==<br /> <br /> === Prerequisites ===<br /> To use Bastion, an administrator must create an account for you.<br /> <br /> Please include the public SSH key you will use to connect to Bastion in your request for an account.<br /> <br /> After creating your account, the administrator will get back to you with your Bastion user name and a proposed alias.<br /> <br /> === Creating the Bastion alias ===<br /> For your convenience, it is recommended to create an alias for the Bastion connection:<syntaxhighlight lang="bash"><br /> alias bssh='ssh -t <YOUR BASTION USERNAME>@bastion.sd4h.ca --'<br /> </syntaxhighlight>After adding the alias, you can connect to (or through) Bastion!<br /> <br /> === Connection types ===<br /> There are three types of connections you can make.<br /> <br /> # Interactive mode<br /> # Non-interactive mode<br /> # Directly to an external, registered server.<br /> In a direct connection, the bastion acts as a transparent pass-through. This is the most convenient, likely and common way you will connect. See the "Connecting to a server" section below.<br /> <br /> <syntaxhighlight lang="bash"><br /> # Interactive mode (signs out when idle)<br /> bssh # SSH into Bastion<br /> help # runs the Bastion 'help' command<br /> <br /> # Non-interactive mode equivalent with '--osh'<br /> bssh --osh help<br /> </syntaxhighlight><br /> <br /> === Permissions ===<br /> Bastion uses the principle of least privileges, meaning that users must be granted explicit access to servers.<br /> <br /> As such, a fresh Bastion user will not have access to anything at first, accesses must be granted by an administrator.<br /> <br /> === Listing your server accesses ===<br /> The <code>selfListAccesses</code> command lists the servers you have access to in Bastion.<br /> <br /> To view your accesses, simply run the command:<syntaxhighlight lang="bash"><br /> bssh --osh selfListAccesses<br /> <br /> # Dear <USERNAME>, you have access to the following servers:<br /> # IP PORT USER ACCESS-BY ADDED-BY ADDED-AT<br /> # -------------- ---- -------------- ---------------------- ---------- ----------<br /> # <SERVER 1 IP> 22 <SERVER 1 USER> <GROUP>(group-member) vrocheleau 2024-09-11<br /> # <SERVER 2 IP> 22 <SERVER 2 USER> personal vrocheleau 2024-09-11<br /> <br /> </syntaxhighlight><br /> <br /> === Connecting to a server ===<br /> You can connect directly to any server that is listed by the <code>selfListAccesses</code> command.<br /> <br /> Taking the example output from the previous section:<syntaxhighlight lang="bash"><br /> # connect to "SERVER 1"<br /> bssh <SERVER 1 USER>@<SERVER 1 IP><br /> <br /> # connect to "SERVER 2"<br /> bssh <SERVER 2 USER>@<SERVER 2 IP><br /> </syntaxhighlight>OpenStack provides name resolution for VMs on the same network. If the Bastion server is on the same OpenStack network as the destination servers, you may use the VM name rather than its IP address. This makes using bssh essentially identical to using ssh (though rather than a destination domain name, the VM name is used). <syntaxhighlight lang="bash"><br /> bssh <SERVER 1 USER>@<SERVER 1 NAME><br /> <br /> </syntaxhighlight><br /> <br /> === Using scp, sftp, rsync through The Bastion ===<br /> More to follow.<br /> <br /> === Access types ===<br /> Coming soon!<br /> <br /> ==== Group ====<br /> <br /> ==== Personal ====</div>

Dbrownlee https://wiki.c3g-app.sd4h.ca/index.php?title=SSH_to_a_server_via_Bastion&diff=181 SSH to a server via Bastion 2025-08-06T21:46:59Z

<p>Dbrownlee: /* Using scp, sftp, rsync */</p> <hr /> <div>== Bastion in a nutshell ==<br /> [https://ovh.github.io/the-bastion/index.html The Bastion] is a single point of entry for SSH connections.<br /> <br /> Some servers in SD4H may not be publicly accessible from the public internet for security reasons.<br /> <br /> Bastion is used as an SSH proxy to connect users to servers.<br /> <br /> == User guide ==<br /> <br /> === Prerequisites ===<br /> To use Bastion, an administrator must create an account for you.<br /> <br /> Please include the public SSH key you will use to connect to Bastion in your request for an account.<br /> <br /> After creating your account, the administrator will get back to you with your Bastion user name and a proposed alias.<br /> <br /> === Creating the Bastion alias ===<br /> For your convenience, it is recommended to create an alias for the Bastion connection:<syntaxhighlight lang="bash"><br /> alias bssh='ssh -t <YOUR BASTION USERNAME>@bastion.sd4h.ca --'<br /> </syntaxhighlight>After adding the alias, you can connect to (or through) Bastion!<br /> <br /> === Connection types ===<br /> There are three types of connections you can make.<br /> <br /> # Interactive mode<br /> # Non-interactive mode<br /> # Directly to an external, registered server.<br /> In a direct connection, the bastion acts as a transparent pass-through. This is the most convenient, likely and common way you will connect. See the "Connecting to a server" section below.<br /> <br /> <syntaxhighlight lang="bash"><br /> # Interactive mode (signs out when idle)<br /> bssh # SSH into Bastion<br /> help # runs the Bastion 'help' command<br /> <br /> # Non-interactive mode equivalent with '--osh'<br /> bssh --osh help<br /> </syntaxhighlight><br /> <br /> === Permissions ===<br /> Bastion uses the principle of least privileges, meaning that users must be granted explicit access to servers.<br /> <br /> As such, a fresh Bastion user will not have access to anything at first, accesses must be granted by an administrator.<br /> <br /> === Listing your server accesses ===<br /> The <code>selfListAccesses</code> command lists the servers you have access to in Bastion.<br /> <br /> To view your accesses, simply run the command:<syntaxhighlight lang="bash"><br /> bssh --osh selfListAccesses<br /> <br /> # Dear <USERNAME>, you have access to the following servers:<br /> # IP PORT USER ACCESS-BY ADDED-BY ADDED-AT<br /> # -------------- ---- -------------- ---------------------- ---------- ----------<br /> # <SERVER 1 IP> 22 <SERVER 1 USER> <GROUP>(group-member) vrocheleau 2024-09-11<br /> # <SERVER 2 IP> 22 <SERVER 2 USER> personal vrocheleau 2024-09-11<br /> <br /> </syntaxhighlight><br /> <br /> === Connecting to a server ===<br /> You can connect directly to any server that is listed by the <code>selfListAccesses</code> command.<br /> <br /> Taking the example output from the previous section:<syntaxhighlight lang="bash"><br /> # connect to "SERVER 1"<br /> bssh <SERVER 1 USER>@<SERVER 1 IP><br /> <br /> # connect to "SERVER 2"<br /> bssh <SERVER 2 USER>@<SERVER 2 IP><br /> </syntaxhighlight>OpenStack provides name resolution for VMs on the same network. If the Bastion server is on the same OpenStack network as the destination servers, you may use the VM name rather than its IP address. This makes using bssh essentially identical to using ssh (though rather than a destination domain name, the VM name is used). <syntaxhighlight lang="bash"><br /> bssh <SERVER 1 USER>@<SERVER 1 NAME><br /> <br /> </syntaxhighlight><br /> <br /> === Using scp, sftp, rsync through The Bastion ===<br /> More to follow.<br /> <br /> === Access types ===<br /> Coming soon!<br /> <br /> ==== Group ====<br /> <br /> ==== Personal ====</div>

Dbrownlee https://wiki.c3g-app.sd4h.ca/index.php?title=SSH_to_a_server_via_Bastion&diff=180 SSH to a server via Bastion 2025-08-06T21:46:39Z

<p>Dbrownlee: </p> <hr /> <div>== Bastion in a nutshell ==<br /> [https://ovh.github.io/the-bastion/index.html The Bastion] is a single point of entry for SSH connections.<br /> <br /> Some servers in SD4H may not be publicly accessible from the public internet for security reasons.<br /> <br /> Bastion is used as an SSH proxy to connect users to servers.<br /> <br /> == User guide ==<br /> <br /> === Prerequisites ===<br /> To use Bastion, an administrator must create an account for you.<br /> <br /> Please include the public SSH key you will use to connect to Bastion in your request for an account.<br /> <br /> After creating your account, the administrator will get back to you with your Bastion user name and a proposed alias.<br /> <br /> === Creating the Bastion alias ===<br /> For your convenience, it is recommended to create an alias for the Bastion connection:<syntaxhighlight lang="bash"><br /> alias bssh='ssh -t <YOUR BASTION USERNAME>@bastion.sd4h.ca --'<br /> </syntaxhighlight>After adding the alias, you can connect to (or through) Bastion!<br /> <br /> === Connection types ===<br /> There are three types of connections you can make.<br /> <br /> # Interactive mode<br /> # Non-interactive mode<br /> # Directly to an external, registered server.<br /> In a direct connection, the bastion acts as a transparent pass-through. This is the most convenient, likely and common way you will connect. See the "Connecting to a server" section below.<br /> <br /> <syntaxhighlight lang="bash"><br /> # Interactive mode (signs out when idle)<br /> bssh # SSH into Bastion<br /> help # runs the Bastion 'help' command<br /> <br /> # Non-interactive mode equivalent with '--osh'<br /> bssh --osh help<br /> </syntaxhighlight><br /> <br /> === Permissions ===<br /> Bastion uses the principle of least privileges, meaning that users must be granted explicit access to servers.<br /> <br /> As such, a fresh Bastion user will not have access to anything at first, accesses must be granted by an administrator.<br /> <br /> === Listing your server accesses ===<br /> The <code>selfListAccesses</code> command lists the servers you have access to in Bastion.<br /> <br /> To view your accesses, simply run the command:<syntaxhighlight lang="bash"><br /> bssh --osh selfListAccesses<br /> <br /> # Dear <USERNAME>, you have access to the following servers:<br /> # IP PORT USER ACCESS-BY ADDED-BY ADDED-AT<br /> # -------------- ---- -------------- ---------------------- ---------- ----------<br /> # <SERVER 1 IP> 22 <SERVER 1 USER> <GROUP>(group-member) vrocheleau 2024-09-11<br /> # <SERVER 2 IP> 22 <SERVER 2 USER> personal vrocheleau 2024-09-11<br /> <br /> </syntaxhighlight><br /> <br /> === Connecting to a server ===<br /> You can connect directly to any server that is listed by the <code>selfListAccesses</code> command.<br /> <br /> Taking the example output from the previous section:<syntaxhighlight lang="bash"><br /> # connect to "SERVER 1"<br /> bssh <SERVER 1 USER>@<SERVER 1 IP><br /> <br /> # connect to "SERVER 2"<br /> bssh <SERVER 2 USER>@<SERVER 2 IP><br /> </syntaxhighlight>OpenStack provides name resolution for VMs on the same network. If the Bastion server is on the same OpenStack network as the destination servers, you may use the VM name rather than its IP address. This makes using bssh essentially identical to using ssh (though rather than a destination domain name, the VM name is used). <syntaxhighlight lang="bash"><br /> bssh <SERVER 1 USER>@<SERVER 1 NAME><br /> <br /> </syntaxhighlight><br /> <br /> === Using scp, sftp, rsync ===<br /> More to follow.<br /> <br /> === Access types ===<br /> Coming soon!<br /> <br /> ==== Group ====<br /> <br /> ==== Personal ====</div>

Dbrownlee https://wiki.c3g-app.sd4h.ca/index.php?title=SSH_to_a_server_via_Bastion&diff=179 SSH to a server via Bastion 2025-08-06T21:44:57Z

<p>Dbrownlee: </p> <hr /> <div>== Bastion in a nutshell ==<br /> [https://ovh.github.io/the-bastion/index.html The Bastion] is a single point of entry for SSH connections.<br /> <br /> Some servers in SD4H may not be publicly accessible from the public internet for security reasons.<br /> <br /> Bastion is used as an SSH proxy to connect users to servers.<br /> <br /> == User guide ==<br /> <br /> === Prerequisites ===<br /> To use Bastion, an administrator must create an account for you.<br /> <br /> Please include the public SSH key you will use to connect to Bastion in your request for an account.<br /> <br /> After creating your account, the administrator will get back to you with your Bastion user name and a proposed alias.<br /> <br /> === Creating the Bastion alias ===<br /> For your convenience, it is recommended to create an alias for the Bastion connection:<syntaxhighlight lang="bash"><br /> alias bssh='ssh -t <YOUR BASTION USERNAME>@bastion.sd4h.ca --'<br /> </syntaxhighlight>After adding the alias, you can connect to (or through) Bastion!<br /> <br /> === Connection types ===<br /> There are three types of connections you can make.<br /> <br /> # Interactive mode<br /> # Non-interactive mode<br /> # Directly to an external, registered server.<br /> In a direct connection, the bastion acts as a transparent pass-through. This is the most convenient, likely and common way you will connect. See the "Connecting to a server" section below.<br /> <br /> <syntaxhighlight lang="bash"><br /> # Interactive mode (signs out when idle)<br /> bssh # SSH into Bastion<br /> help # runs the Bastion 'help' command<br /> <br /> # Non-interactive mode equivalent with '--osh'<br /> bssh --osh help<br /> </syntaxhighlight><br /> <br /> === Permissions ===<br /> Bastion uses the principle of least privileges, meaning that users must be granted explicit access to servers.<br /> <br /> As such, a fresh Bastion user will not have access to anything at first, accesses must be granted by an administrator.<br /> <br /> === Listing your server accesses ===<br /> The <code>selfListAccesses</code> command lists the servers you have access to in Bastion.<br /> <br /> To view your accesses, simply run the command:<syntaxhighlight lang="bash"><br /> bssh --osh selfListAccesses<br /> <br /> # Dear <USERNAME>, you have access to the following servers:<br /> # IP PORT USER ACCESS-BY ADDED-BY ADDED-AT<br /> # -------------- ---- -------------- ---------------------- ---------- ----------<br /> # <SERVER 1 IP> 22 <SERVER 1 USER> <GROUP>(group-member) vrocheleau 2024-09-11<br /> # <SERVER 2 IP> 22 <SERVER 2 USER> personal vrocheleau 2024-09-11<br /> <br /> </syntaxhighlight><br /> <br /> === Connecting to a server ===<br /> You can connect directly to any server that is listed by the <code>selfListAccesses</code> command.<br /> <br /> Taking the example output from the previous section:<syntaxhighlight lang="bash"><br /> # connect to "SERVER 1"<br /> bssh <SERVER 1 USER>@<SERVER 1 IP><br /> <br /> # connect to "SERVER 2"<br /> bssh <SERVER 2 USER>@<SERVER 2 IP><br /> </syntaxhighlight>OpenStack provides name resolution for VMs on the same network. If the Bastion server is on the same OpenStack network as the destination servers, you may use the VM name rather than its IP address. This makes using bssh essentially identical to using ssh (though rather than a destination domain name, the VM name is used). <syntaxhighlight lang="bash"><br /> bssh <SERVER 1 USER>@<SERVER 1 NAME><br /> <br /> </syntaxhighlight><br /> <br /> === Access types ===<br /> Coming soon!<br /> <br /> ==== Group ====<br /> <br /> ==== Personal ====</div>

Dbrownlee https://wiki.c3g-app.sd4h.ca/index.php?title=SSH_to_a_server_via_Bastion&diff=178 SSH to a server via Bastion 2025-08-06T21:43:33Z

<p>Dbrownlee: /* Connection types */</p> <hr /> <div>== Bastion in a nutshell ==<br /> [https://ovh.github.io/the-bastion/index.html The Bastion] is a single point of entry for SSH connections.<br /> <br /> Some servers in SD4H may not be publicly accessible from the public internet for security reasons.<br /> <br /> Bastion is used as an SSH proxy to connect users to servers.<br /> <br /> == User guide ==<br /> <br /> === Prerequisites ===<br /> To use Bastion, an administrator must create an account for you.<br /> <br /> Please include the public SSH key you will use to connect to Bastion in your request for an account.<br /> <br /> After creating your account, the administrator will get back to you with your Bastion user name and a proposed alias.<br /> <br /> === Creating the Bastion alias ===<br /> For your convenience, it is recommended to create an alias for the Bastion connection:<syntaxhighlight lang="bash"><br /> alias bssh='ssh -t <YOUR BASTION USERNAME>@bastion.sd4h.ca --'<br /> </syntaxhighlight>After adding the alias, you can connect to (or through) Bastion!<br /> <br /> === Connection types ===<br /> There are three types of connections you can make.<br /> <br /> # Interactive mode<br /> # Non-interactive mode<br /> # Directly to a registered server.<br /> In a direct connection, the bastion acts as a pass-through. This is the most convenient, likely and common way you will use connect. See the "Connecting to a server" section below.<br /> <br /> <syntaxhighlight lang="bash"><br /> # Interactive mode (signs out when idle)<br /> bssh # SSH into Bastion<br /> help # runs the Bastion 'help' command<br /> <br /> # Non-interactive mode equivalent with '--osh'<br /> bssh --osh help<br /> </syntaxhighlight><br /> <br /> === Permissions ===<br /> Bastion uses the principle of least privileges, meaning that users must be granted explicit access to servers.<br /> <br /> As such, a fresh Bastion user will not have access to anything at first, accesses must be granted by an administrator.<br /> <br /> === Listing your server accesses ===<br /> The <code>selfListAccesses</code> command lists the servers you have access to in Bastion.<br /> <br /> To view your accesses, simply run the command:<syntaxhighlight lang="bash"><br /> bssh --osh selfListAccesses<br /> <br /> # Dear <USERNAME>, you have access to the following servers:<br /> # IP PORT USER ACCESS-BY ADDED-BY ADDED-AT<br /> # -------------- ---- -------------- ---------------------- ---------- ----------<br /> # <SERVER 1 IP> 22 <SERVER 1 USER> <GROUP>(group-member) vrocheleau 2024-09-11<br /> # <SERVER 2 IP> 22 <SERVER 2 USER> personal vrocheleau 2024-09-11<br /> <br /> </syntaxhighlight><br /> <br /> === Connecting to a server ===<br /> You can connect directly to any server that is listed by the <code>selfListAccesses</code> command.<br /> <br /> Taking the example output from the previous section:<syntaxhighlight lang="bash"><br /> # connect to "SERVER 1"<br /> bssh <SERVER 1 USER>@<SERVER 1 IP><br /> <br /> # connect to "SERVER 2"<br /> bssh <SERVER 2 USER>@<SERVER 2 IP><br /> </syntaxhighlight>OpenStack provides name resolution for VMs on the same network. If the Bastion server is on the same OpenStack network as the destination servers, you may use the VM name rather than its IP address. This makes using bssh essentially identical to using ssh (though rather than a destination domain name, the VM name is used). <syntaxhighlight lang="bash"><br /> bssh <SERVER 1 USER>@<SERVER 1 NAME><br /> <br /> </syntaxhighlight><br /> <br /> === Access types ===<br /> Coming soon!<br /> <br /> ==== Group ====<br /> <br /> ==== Personal ====</div>

Dbrownlee https://wiki.c3g-app.sd4h.ca/index.php?title=SSH_to_a_server_via_Bastion&diff=177 SSH to a server via Bastion 2025-08-06T21:38:50Z

<p>Dbrownlee: </p> <hr /> <div>== Bastion in a nutshell ==<br /> [https://ovh.github.io/the-bastion/index.html The Bastion] is a single point of entry for SSH connections.<br /> <br /> Some servers in SD4H may not be publicly accessible from the public internet for security reasons.<br /> <br /> Bastion is used as an SSH proxy to connect users to servers.<br /> <br /> == User guide ==<br /> <br /> === Prerequisites ===<br /> To use Bastion, an administrator must create an account for you.<br /> <br /> Please include the public SSH key you will use to connect to Bastion in your request for an account.<br /> <br /> After creating your account, the administrator will get back to you with your Bastion user name and a proposed alias.<br /> <br /> === Creating the Bastion alias ===<br /> For your convenience, it is recommended to create an alias for the Bastion connection:<syntaxhighlight lang="bash"><br /> alias bssh='ssh -t <YOUR BASTION USERNAME>@bastion.sd4h.ca --'<br /> </syntaxhighlight>After adding the alias, you can connect to (or through) Bastion!<br /> <br /> === Connection types ===<br /> There are three types of connections you can make.<br /> <br /> # Interactive mode<br /> # Non-interactive mode<br /> # Directly to a registered server (the bastion acts as a pass-through). <br /> <syntaxhighlight lang="bash"><br /> # Interactive mode (signs out when idle)<br /> bssh # SSH into Bastion<br /> help # runs the Bastion 'help' command<br /> <br /> # Non-interactive mode equivalent with '--osh'<br /> bssh --osh help<br /> </syntaxhighlight><br /> <br /> === Permissions ===<br /> Bastion uses the principle of least privileges, meaning that users must be granted explicit access to servers.<br /> <br /> As such, a fresh Bastion user will not have access to anything at first, accesses must be granted by an administrator.<br /> <br /> === Listing your server accesses ===<br /> The <code>selfListAccesses</code> command lists the servers you have access to in Bastion.<br /> <br /> To view your accesses, simply run the command:<syntaxhighlight lang="bash"><br /> bssh --osh selfListAccesses<br /> <br /> # Dear <USERNAME>, you have access to the following servers:<br /> # IP PORT USER ACCESS-BY ADDED-BY ADDED-AT<br /> # -------------- ---- -------------- ---------------------- ---------- ----------<br /> # <SERVER 1 IP> 22 <SERVER 1 USER> <GROUP>(group-member) vrocheleau 2024-09-11<br /> # <SERVER 2 IP> 22 <SERVER 2 USER> personal vrocheleau 2024-09-11<br /> <br /> </syntaxhighlight><br /> <br /> === Connecting to a server ===<br /> You can connect directly to any server that is listed by the <code>selfListAccesses</code> command.<br /> <br /> Taking the example output from the previous section:<syntaxhighlight lang="bash"><br /> # connect to "SERVER 1"<br /> bssh <SERVER 1 USER>@<SERVER 1 IP><br /> <br /> # connect to "SERVER 2"<br /> bssh <SERVER 2 USER>@<SERVER 2 IP><br /> </syntaxhighlight>OpenStack provides name resolution for VMs on the same network. If the Bastion server is on the same OpenStack network as the destination servers, you may use the VM name rather than its IP address. This makes using bssh essentially identical to using ssh (though rather than a destination domain name, the VM name is used). <syntaxhighlight lang="bash"><br /> bssh <SERVER 1 USER>@<SERVER 1 NAME><br /> <br /> </syntaxhighlight><br /> <br /> === Access types ===<br /> Coming soon!<br /> <br /> ==== Group ====<br /> <br /> ==== Personal ====</div>

Dbrownlee https://wiki.c3g-app.sd4h.ca/index.php?title=SSH_to_a_server_via_Bastion&diff=176 SSH to a server via Bastion 2025-08-06T21:34:39Z

<p>Dbrownlee: </p> <hr /> <div>== Bastion in a nutshell ==<br /> [https://ovh.github.io/the-bastion/index.html The Bastion] is a single point of entry for SSH connections.<br /> <br /> Some servers in SD4H may not be publicly accessible from the public internet for security reasons.<br /> <br /> Bastion is used as an SSH proxy to connect users to servers.<br /> <br /> == User guide ==<br /> <br /> === Prerequisites ===<br /> To use Bastion, an administrator must create an account for you.<br /> <br /> Please include the public SSH key you will use to connect to Bastion in your request for an account.<br /> <br /> After creating your account, the administrator will get back to you with your Bastion user name and a proposed alias.<br /> <br /> === Creating the Bastion alias ===<br /> For your convenience, it is recommended to create an alias for the Bastion connection:<syntaxhighlight lang="bash"><br /> alias bssh='ssh -t <YOUR BASTION USERNAME>@bastion.sd4h.ca --'<br /> </syntaxhighlight>After adding the alias, you can connect to Bastion!<syntaxhighlight lang="bash"><br /> # Interactive mode (signs out when idle)<br /> bssh # SSH into Bastion<br /> help # runs the Bastion 'help' command<br /> <br /> # Non-interactive mode equivalent with '--osh'<br /> bssh --osh help<br /> </syntaxhighlight><br /> <br /> === Permissions ===<br /> Bastion uses the principle of least privileges, meaning that users must be granted explicit access to servers.<br /> <br /> As such, a fresh Bastion user will not have access to anything at first, accesses must be granted by an administrator.<br /> <br /> === Listing your server accesses ===<br /> The <code>selfListAccesses</code> command lists the servers you have access to in Bastion.<br /> <br /> To view your accesses, simply run the command:<syntaxhighlight lang="bash"><br /> bssh --osh selfListAccesses<br /> <br /> # Dear <USERNAME>, you have access to the following servers:<br /> # IP PORT USER ACCESS-BY ADDED-BY ADDED-AT<br /> # -------------- ---- -------------- ---------------------- ---------- ----------<br /> # <SERVER 1 IP> 22 <SERVER 1 USER> <GROUP>(group-member) vrocheleau 2024-09-11<br /> # <SERVER 2 IP> 22 <SERVER 2 USER> personal vrocheleau 2024-09-11<br /> <br /> </syntaxhighlight><br /> <br /> === Connecting to a server ===<br /> You can connect directly to any server that is listed by the <code>selfListAccesses</code> command.<br /> <br /> Taking the example output from the previous section:<syntaxhighlight lang="bash"><br /> # connect to "SERVER 1"<br /> bssh <SERVER 1 USER>@<SERVER 1 IP><br /> <br /> # connect to "SERVER 2"<br /> bssh <SERVER 2 USER>@<SERVER 2 IP><br /> </syntaxhighlight>OpenStack provides name resolution for VMs on the same network. If the Bastion server is on the same OpenStack network as the destination servers, you may use the VM name rather than its IP address. This makes using bssh essentially identical to using ssh (though rather than a destination domain name, the VM name is used). <syntaxhighlight lang="bash"><br /> bssh <SERVER 1 USER>@<SERVER 1 NAME><br /> <br /> </syntaxhighlight><br /> <br /> === Access types ===<br /> Coming soon!<br /> <br /> ==== Group ====<br /> <br /> ==== Personal ====</div>

Dbrownlee https://wiki.c3g-app.sd4h.ca/index.php?title=SSH_to_a_server_via_Bastion&diff=175 SSH to a server via Bastion 2025-08-06T21:26:19Z

<p>Dbrownlee: </p> <hr /> <div>== Bastion in a nutshell ==<br /> [https://ovh.github.io/the-bastion/index.html The Bastion] is a single point of entry for SSH connections.<br /> <br /> Some servers in SD4H may not be publicly accessible from the public internet for security reasons.<br /> <br /> Bastion is used as an SSH proxy to connect users to servers.<br /> <br /> == User guide ==<br /> <br /> === Prerequisites ===<br /> To use Bastion, an administrator must create an account for you.<br /> <br /> Please include the public SSH key you will use to connect to Bastion in your request for an account.<br /> <br /> After creating your account, the administrator will get back to you with your Bastion user name.<br /> <br /> === Creating the Bastion alias ===<br /> For your convenience, it is recommended to create an alias for the Bastion connection:<syntaxhighlight lang="bash"><br /> alias bssh='ssh -t <YOUR BASTION USERNAME>@bastion.sd4h.ca --'<br /> </syntaxhighlight>After adding the alias, you can connect to Bastion!<syntaxhighlight lang="bash"><br /> # Interactive mode (signs out when idle)<br /> bssh # SSH into Bastion<br /> help # runs the Bastion 'help' command<br /> <br /> # Non-interactive mode equivalent with '--osh'<br /> bssh --osh help<br /> </syntaxhighlight><br /> <br /> === Permissions ===<br /> Bastion uses the principle of least privileges, meaning that users must be granted explicit access to servers.<br /> <br /> As such, a fresh Bastion user will not have access to anything at first, accesses must be granted by an administrator.<br /> <br /> === Listing your server accesses ===<br /> The <code>selfListAccesses</code> command lists the servers you have access to in Bastion.<br /> <br /> To view your accesses, simply run the command:<syntaxhighlight lang="bash"><br /> bssh --osh selfListAccesses<br /> <br /> # Dear <USERNAME>, you have access to the following servers:<br /> # IP PORT USER ACCESS-BY ADDED-BY ADDED-AT<br /> # -------------- ---- -------------- ---------------------- ---------- ----------<br /> # <SERVER 1 IP> 22 <SERVER 1 USER> <GROUP>(group-member) vrocheleau 2024-09-11<br /> # <SERVER 2 IP> 22 <SERVER 2 USER> personal vrocheleau 2024-09-11<br /> <br /> </syntaxhighlight><br /> <br /> === Connecting to a server ===<br /> You can connect directly to any server that is listed by the <code>selfListAccesses</code> command.<br /> <br /> Taking the example output from the previous section:<syntaxhighlight lang="bash"><br /> # connect to "SERVER 1"<br /> bssh <SERVER 1 USER>@<SERVER 1 IP><br /> <br /> # connect to "SERVER 2"<br /> bssh <SERVER 2 USER>@<SERVER 2 IP><br /> </syntaxhighlight>OpenStack provides name resolution for VMs on the same network. If the Bastion server is on the same OpenStack network as the destination servers, you may use the VM name rather than its IP address. This makes using bssh essentially identical to using ssh (though rather than a destination domain name, the VM name is used). <syntaxhighlight lang="bash"><br /> bssh <SERVER 1 USER>@<SERVER 1 NAME><br /> <br /> </syntaxhighlight><br /> <br /> === Access types ===<br /> Coming soon!<br /> <br /> ==== Group ====<br /> <br /> ==== Personal ====</div>

Dbrownlee https://wiki.c3g-app.sd4h.ca/index.php?title=SSH_to_a_server_via_Bastion&diff=174 SSH to a server via Bastion 2025-08-06T21:21:32Z

<p>Dbrownlee: </p> <hr /> <div>== Bastion in a nutshell ==<br /> [https://ovh.github.io/the-bastion/index.html The Bastion] is a single point of entry for SSH connections.<br /> <br /> Some servers in SD4H may not be publicly accessible from the public internet for security reasons.<br /> <br /> Bastion is used as an SSH proxy to connect users to servers.<br /> <br /> == User guide ==<br /> <br /> === Prerequisites ===<br /> To use Bastion, an administrator must create an account for you.<br /> <br /> Please include the public SSH key you will use to connect to Bastion in your request for an account.<br /> <br /> After creating your account, the administrator will get back to you with your Bastion user name.<br /> <br /> === Creating the Bastion alias ===<br /> For your convenience, it is recommended to create an alias for the Bastion connection:<syntaxhighlight lang="bash"><br /> alias bssh='ssh -t <YOUR BASTION USERNAME>@bastion.sd4h.ca --'<br /> </syntaxhighlight>After adding the alias, you can connect to Bastion!<syntaxhighlight lang="bash"><br /> # Interactive mode (signs out when idle)<br /> bssh # SSH into Bastion<br /> help # runs the Bastion 'help' command<br /> <br /> # Non-interactive mode equivalent with '--osh'<br /> bssh --osh help<br /> </syntaxhighlight><br /> <br /> === Permissions ===<br /> Bastion uses the principle of least privileges, meaning that users must be granted explicit access to servers.<br /> <br /> As such, a fresh Bastion user will not have access to anything at first, accesses must be granted by an administrator.<br /> <br /> === Listing your server accesses ===<br /> The <code>selfListAccesses</code> command lists the servers you have access to in Bastion.<br /> <br /> To view your accesses, simply run the command:<syntaxhighlight lang="bash"><br /> bssh --osh selfListAccesses<br /> <br /> # Dear <USERNAME>, you have access to the following servers:<br /> # IP PORT USER ACCESS-BY ADDED-BY ADDED-AT<br /> # -------------- ---- -------------- ---------------------- ---------- ----------<br /> # <SERVER 1 IP> 22 <SERVER 1 USER> <GROUP>(group-member) vrocheleau 2024-09-11<br /> # <SERVER 2 IP> 22 <SERVER 2 USER> personal vrocheleau 2024-09-11<br /> <br /> </syntaxhighlight><br /> <br /> === Connecting to a server ===<br /> You can connect directly to any server that is listed by the <code>selfListAccesses</code> command.<br /> <br /> Taking the example output from the previous section:<syntaxhighlight lang="bash"><br /> # connect to "SERVER 1"<br /> bssh <SERVER 1 USER>@<SERVER 1 IP><br /> <br /> # connect to "SERVER 2"<br /> bssh <SERVER 2 USER>@<SERVER 2 IP><br /> </syntaxhighlight>OpenStack provides name resolution for VMs on the same network. If the bastion server is on the same OpenStack network as the destination servers, you may use the VM name rather than its IP address. This makes using bssh essentially identical to using ssh (though rather than a domain name, the VM name is used). <syntaxhighlight lang="bash"><br /> bssh <SERVER 1 USER>@<SERVER 1 NAME><br /> <br /> </syntaxhighlight><br /> <br /> === Access types ===<br /> Coming soon!<br /> <br /> ==== Group ====<br /> <br /> ==== Personal ====</div>

Dbrownlee https://wiki.c3g-app.sd4h.ca/index.php?title=SSH_to_a_server_via_Bastion&diff=173 SSH to a server via Bastion 2025-08-06T21:13:49Z

<p>Dbrownlee: </p> <hr /> <div>== Bastion in a nutshell ==<br /> [https://ovh.github.io/the-bastion/index.html The Bastion] is a single point of entry for SSH connections.<br /> <br /> Some servers in SD4H may not be publicly accessible from the public internet for security reasons.<br /> <br /> Bastion is used as an SSH proxy to connect users to servers.<br /> <br /> == User guide ==<br /> <br /> === Prerequisites ===<br /> To use Bastion, an administrator must create an account for you.<br /> <br /> Please include the public SSH key you will use to connect to Bastion in your request for an account.<br /> <br /> After creating your account, the administrator will get back to you with your Bastion user name.<br /> <br /> === Creating the Bastion alias ===<br /> For your convenience, it is recommended to create an alias for the Bastion connection:<syntaxhighlight lang="bash"><br /> alias bssh='ssh -t <YOUR BASTION USERNAME>@bastion.sd4h.ca --'<br /> </syntaxhighlight>After adding the alias, you can connect to Bastion!<syntaxhighlight lang="bash"><br /> # Interactive mode (signs out when idle)<br /> bssh # SSH into Bastion<br /> help # runs the Bastion 'help' command<br /> <br /> # Non-interactive mode equivalent with '--osh'<br /> bssh --osh help<br /> </syntaxhighlight><br /> <br /> === Permissions ===<br /> Bastion uses the principle of least privileges, meaning that users must be granted explicit access to servers.<br /> <br /> As such, a fresh Bastion user will not have access to anything at first, accesses must be granted by an administrator.<br /> <br /> === Listing your server accesses ===<br /> The <code>selfListAccesses</code> command lists the servers you have access to in Bastion.<br /> <br /> To view your accesses, simply run the command:<syntaxhighlight lang="bash"><br /> bssh --osh selfListAccesses<br /> <br /> # Dear <USERNAME>, you have access to the following servers:<br /> # IP PORT USER ACCESS-BY ADDED-BY ADDED-AT<br /> # -------------- ---- -------------- ---------------------- ---------- ----------<br /> # <SERVER 1 IP> 22 <SERVER 1 USER> <GROUP>(group-member) vrocheleau 2024-09-11<br /> # <SERVER 2 IP> 22 <SERVER 2 USER> personal vrocheleau 2024-09-11<br /> <br /> </syntaxhighlight><br /> <br /> === Connecting to a server ===<br /> You can connect directly to any server that is listed by the <code>selfListAccesses</code> command.<br /> <br /> Taking the example output from the previous section:<syntaxhighlight lang="bash"><br /> # connect to "SERVER 1"<br /> bssh <SERVER 1 USER>@<SERVER 1 IP><br /> <br /> # connect to "SERVER 2"<br /> bssh <SERVER 2 USER>@<SERVER 2 IP><br /> </syntaxhighlight>You may avoid using IP addresses by creating a host name in your ~/.ssh/config such as:<syntaxhighlight lang="bash"><br /> Host <SERVER 1 NAME><br /> HostName <SERVER 1 IP><br /> </syntaxhighlight>This allows you to connect through bastion directly to any server with:<syntaxhighlight lang="bash"><br /> # Connect to a server, using the ~/.ssh/config host name.<br /> # This makes using bssh essentially identical to using ssh. <br /> bssh <SERVER 1 USER>@<SERVER 1 NAME><br /> </syntaxhighlight><br /> <br /> === Access types ===<br /> Coming soon!<br /> <br /> ==== Group ====<br /> <br /> ==== Personal ====</div>

Dbrownlee https://wiki.c3g-app.sd4h.ca/index.php?title=Backing_up_Object_Store&diff=170 Backing up Object Store 2025-06-03T21:37:23Z

<p>Dbrownlee: Restore procedure clean-up.</p> <hr /> <div>__FORCETOC__<br /> <br /> Object Store data, while stored redundantly via Ceph, is not backed up by default. Object Store buckets are backed up to the TSM tape system upon request only by following the procedures listed on this page.<br /> <br /> == Content and policies ==<br /> The following are the default contents and policies of requested backups:<br /> <br /> What is in the backup?<br /> * Only the bucket data is backed up. We are not currently backing up the IAM policies of the buckets or objects.<br /> * Only the current version of the data is seen by the backup system. Object chunks or versioned objects are not seen by the backups system.<br /> What is the backup policy?<br /> * Backups are run on a daily basis.<br /> * The current object and one modified version of object are kept (this is different than full bucket versioning). <br /> * The modified version is kept for 6 month - after that period only the current object is kept. <br /> * Deleted objects are kept for 6 months.<br /> <br /> = Backup Procedure =<br /> <br /> Please follow this procedure to request backups of your buckets. <br /> <br /> ==Email the list of buckets==<br /> <br /> Send a list of buckets to be backed up to [mailto:juno@calculquebec.ca sd4h support] with the name and ID of the project where the buckets live.<br /> <br /> ==Give us permission==<br /> <br /> An IAM policy statement must be applied to '''all the buckets''' you want to backup so the TSM robot user in charge of the backup can access them. This can be done with the [https://awscli.amazonaws.com/v2/documentation/api/latest/index.html aws cli].<br /> <br /> First, ensure that <code>my-bucket</code> currently has no IAM policy. Check bucket <code>my-bucket</code> using profile <code>my-profile</code> (as defined in ~/.aws/config and ~/.aws/credentials files) with: <br /> <br /> <pre>$aws s3api get-bucket-policy --profile c3g-data-repos --bucket my-bucket<br /> <br /> An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist<br /> </pre><br /> <br /> If that command returns something, the new policy statements must be added to the existing policy (which is not covered here).<br /> <br /> The following policy.json needs to be applied.<br /> <div class="filename">'''File :''' policy.json </div><br /> <syntaxhighlight lang="json" file="my-policy.json"><br /> {<br /> "Statement": [<br /> {<br /> "Effect": "Allow",<br /> "Principal": {"AWS": ["arn:aws:iam:::user/tsm"]},<br /> "Action": [<br /> "s3:ListBucket",<br /> "s3:GetObject"<br /> ],<br /> "Resource": [<br /> "arn:aws:s3:::my-bucket/*",<br /> "arn:aws:s3:::my-bucket"<br /> ]<br /> }<br /> ]<br /> }<br /> </syntaxhighlight><br /> <br /> Load the policy onto the bucket <code>my-bucket</code> using the profile <code>my-profile:</code><br /> <br /> <pre><br /> $aws s3api put-bucket-policy --policy file://my-policy.json --profile my-profile --bucket my-bucket<br /> </pre><br /> <br /> == Confirm IAM policy applied ==<br /> As we did before, request the bucket's IAM policy, ensuring that the contents of policy.json are listed.<br /> $aws s3api get-bucket-policy --profile c3g-data-repos --bucket my-bucket<br /> <br /> =Restore Procedure=<br /> <br /> ==List bucket==<br /> <br /> Send us list of buckets or object to restore to [mailto:juno@calculquebec.ca sd4h support]. <br /> <br /> ==Give us permission==<br /> <br /> For each bucket you want to be restored, you will be asked to create a bucket with the <code>-restore</code> suffix. <br /> <br /> Following the previous procedure, apply this restore IAM policy. <br /> <div class="filename">'''File :''' policy.json </div><br /> <syntaxhighlight lang="json" file="my-policy.json"><br /> {<br /> "Statement": [<br /> {<br /> "Effect": "Allow",<br /> "Principal": {"AWS": ["arn:aws:iam:::user/tsm"]},<br /> "Action": [<br /> "s3:ListBucket",<br /> "s3:GetObject",<br /> "s3:PutObject",<br /> "s3:PutObjectAcl",<br /> "s3:AbortMultipartUpload"<br /> ],<br /> "Resource": [<br /> "arn:aws:s3:::my-bucket-restore/*",<br /> "arn:aws:s3:::my-bucket-restore"<br /> ]<br /> }<br /> ]<br /> }<br /> </syntaxhighlight><br /> <br /> Once done, we will restore your data to the <code>*-restore</code> buckets.</div>

Dbrownlee https://wiki.c3g-app.sd4h.ca/index.php?title=Backing_up_Object_Store&diff=169 Backing up Object Store 2025-06-03T20:30:43Z

<p>Dbrownlee: /* Confirm policy applied */</p> <hr /> <div>__FORCETOC__<br /> <br /> Object Store data, while stored redundantly via Ceph, is not backed up. Object Store buckets are backed up to the TSM tape system upon request only by following the procedures listed on this page.<br /> <br /> The following contents and policies apply to backups by default:<br /> <br /> What is in the backup?<br /> * Only the bucket data is backed up. We are not currently backing up the IAM policies of the buckets or objects.<br /> * Only the current version of the data is seen by the backup system. Object chunks or versioned objects are not seen by the backups system.<br /> What is the backup policy?<br /> * Backups are run on a daily basis.<br /> * The current object and one modified version of object are kept (this is different than full bucket versioning). <br /> * The modified version is kept for 6 month - after that period only the current object is kept. <br /> * Deleted objects are kept for 6 months.<br /> <br /> = Backup Procedure =<br /> <br /> Please follow this procedure to request backups of your buckets. <br /> <br /> ==Email the list of buckets==<br /> <br /> Send a list of buckets to backup to [mailto:juno@calculquebec.ca sd4h support] with the name and ID of the project where the buckets live.<br /> <br /> ==Give us permission==<br /> <br /> An iam policy statement must be applied to '''all the buckets''' you want to backup so the TSM robot user in charge of the backup can access them. This can be done with the [https://awscli.amazonaws.com/v2/documentation/api/latest/index.html aws cli].<br /> <br /> First, ensure that <code>my-bucket</code> currently has no policy. Check bucket <code>my-bucket</code> using profile <code>my-profile</code> (as defined in ~/.aws/config and ~/.aws/credentials files): <br /> <br /> <pre>$aws s3api get-bucket-policy --profile c3g-data-repos --bucket my-bucket<br /> <br /> An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist<br /> </pre><br /> <br /> If that command returns something, the new policy statements must be added to the existing policy (which is not covered here).<br /> <br /> The following policy.json needs to be applied.<br /> <div class="filename">'''File :''' policy.json </div><br /> <syntaxhighlight lang="json" file="my-policy.json"><br /> {<br /> "Statement": [<br /> {<br /> "Effect": "Allow",<br /> "Principal": {"AWS": ["arn:aws:iam:::user/tsm"]},<br /> "Action": [<br /> "s3:ListBucket",<br /> "s3:GetObject"<br /> ],<br /> "Resource": [<br /> "arn:aws:s3:::my-bucket/*",<br /> "arn:aws:s3:::my-bucket"<br /> ]<br /> }<br /> ]<br /> }<br /> </syntaxhighlight><br /> <br /> Load the policy onto the bucket <code>my-bucket</code> using the profile <code>my-profile</code><br /> <br /> <pre><br /> $aws s3api put-bucket-policy --policy file://my-policy.json --profile my-profile --bucket my-bucket<br /> </pre><br /> <br /> == Confirm IAM policy applied ==<br /> As we did before, request the bucket's IAM policy, ensuring that the contents of policy.json are listed.<br /> $aws s3api get-bucket-policy --profile c3g-data-repos --bucket my-bucket<br /> <br /> =Restore Procedure=<br /> <br /> ==List bucket==<br /> <br /> Send us list of buckets or object to restore to [mailto:juno@calculquebec.ca sd4h support]. <br /> <br /> ==Give us permission==<br /> <br /> You will be asked to create a bucket for each bucket you want to restore to restore with the <code>-restore</code> suffix. <br /> <div class="filename">'''File :''' policy.json </div><br /> <syntaxhighlight lang="json" file="my-policy.json"><br /> {<br /> "Statement": [<br /> {<br /> "Effect": "Allow",<br /> "Principal": {"AWS": ["arn:aws:iam:::user/tsm"]},<br /> "Action": [<br /> "s3:ListBucket",<br /> "s3:GetObject",<br /> "s3:PutObject",<br /> "s3:PutObjectAcl",<br /> "s3:AbortMultipartUpload"<br /> ],<br /> "Resource": [<br /> "arn:aws:s3:::my-bucket-restore/*",<br /> "arn:aws:s3:::my-bycket-restore"<br /> ]<br /> }<br /> ]<br /> }<br /> </syntaxhighlight><br /> <br /> Once it is done we will restore you data to that folder.</div>

Dbrownlee https://wiki.c3g-app.sd4h.ca/index.php?title=Backing_up_Object_Store&diff=168 Backing up Object Store 2025-06-03T20:28:52Z

<p>Dbrownlee: Confirm policy applied section added.</p> <hr /> <div>__FORCETOC__<br /> <br /> Object Store data, while stored redundantly via Ceph, is not backed up. Object Store buckets are backed up to the TSM tape system upon request only by following the procedures listed on this page.<br /> <br /> The following contents and policies apply to backups by default:<br /> <br /> What is in the backup?<br /> * Only the bucket data is backed up. We are not currently backing up the IAM policies of the buckets or objects.<br /> * Only the current version of the data is seen by the backup system. Object chunks or versioned objects are not seen by the backups system.<br /> What is the backup policy?<br /> * Backups are run on a daily basis.<br /> * The current object and one modified version of object are kept (this is different than full bucket versioning). <br /> * The modified version is kept for 6 month - after that period only the current object is kept. <br /> * Deleted objects are kept for 6 months.<br /> <br /> = Backup Procedure =<br /> <br /> Please follow this procedure to request backups of your buckets. <br /> <br /> ==Email the list of buckets==<br /> <br /> Send a list of buckets to backup to [mailto:juno@calculquebec.ca sd4h support] with the name and ID of the project where the buckets live.<br /> <br /> ==Give us permission==<br /> <br /> An iam policy statement must be applied to '''all the buckets''' you want to backup so the TSM robot user in charge of the backup can access them. This can be done with the [https://awscli.amazonaws.com/v2/documentation/api/latest/index.html aws cli].<br /> <br /> First, ensure that <code>my-bucket</code> currently has no policy. Check bucket <code>my-bucket</code> using profile <code>my-profile</code> (as defined in ~/.aws/config and ~/.aws/credentials files): <br /> <br /> <pre>$aws s3api get-bucket-policy --profile c3g-data-repos --bucket my-bucket<br /> <br /> An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist<br /> </pre><br /> <br /> If that command returns something, the new policy statements must be added to the existing policy (which is not covered here).<br /> <br /> The following policy.json needs to be applied.<br /> <div class="filename">'''File :''' policy.json </div><br /> <syntaxhighlight lang="json" file="my-policy.json"><br /> {<br /> "Statement": [<br /> {<br /> "Effect": "Allow",<br /> "Principal": {"AWS": ["arn:aws:iam:::user/tsm"]},<br /> "Action": [<br /> "s3:ListBucket",<br /> "s3:GetObject"<br /> ],<br /> "Resource": [<br /> "arn:aws:s3:::my-bucket/*",<br /> "arn:aws:s3:::my-bucket"<br /> ]<br /> }<br /> ]<br /> }<br /> </syntaxhighlight><br /> <br /> Load the policy onto the bucket <code>my-bucket</code> using the profile <code>my-profile</code><br /> <br /> <pre><br /> $aws s3api put-bucket-policy --policy file://my-policy.json --profile my-profile --bucket my-bucket<br /> </pre><br /> <br /> == Confirm policy applied ==<br /> As we did before, request the bucket's IAM policy, ensuring that the contents of policy.json are listed.<br /> $aws s3api get-bucket-policy --profile c3g-data-repos --bucket my-bucket<br /> <br /> =Restore Procedure=<br /> <br /> ==List bucket==<br /> <br /> Send us list of buckets or object to restore to [mailto:juno@calculquebec.ca sd4h support]. <br /> <br /> ==Give us permission==<br /> <br /> You will be asked to create a bucket for each bucket you want to restore to restore with the <code>-restore</code> suffix. <br /> <div class="filename">'''File :''' policy.json </div><br /> <syntaxhighlight lang="json" file="my-policy.json"><br /> {<br /> "Statement": [<br /> {<br /> "Effect": "Allow",<br /> "Principal": {"AWS": ["arn:aws:iam:::user/tsm"]},<br /> "Action": [<br /> "s3:ListBucket",<br /> "s3:GetObject",<br /> "s3:PutObject",<br /> "s3:PutObjectAcl",<br /> "s3:AbortMultipartUpload"<br /> ],<br /> "Resource": [<br /> "arn:aws:s3:::my-bucket-restore/*",<br /> "arn:aws:s3:::my-bycket-restore"<br /> ]<br /> }<br /> ]<br /> }<br /> </syntaxhighlight><br /> <br /> Once it is done we will restore you data to that folder.</div>

Dbrownlee https://wiki.c3g-app.sd4h.ca/index.php?title=Backing_up_Object_Store&diff=167 Backing up Object Store 2025-06-03T19:36:40Z

<p>Dbrownlee: Slight re-wording of first para. Minor spelling corrections.</p> <hr /> <div>__FORCETOC__<br /> <br /> Object Store data, while stored redundantly via Ceph, is not backed up. Object Store buckets are backed up to the TSM tape system upon request only by following the procedures listed on this page.<br /> <br /> The following contents and policies apply to backups by default:<br /> <br /> What is in the backup?<br /> * Only the bucket data is backed up. We are not currently backing up the IAM policies of the buckets or objects.<br /> * Only the current version of the data is seen by the backup system. Object chunks or versioned objects are not seen by the backups system.<br /> What is the backup policy?<br /> * Backups are run on a daily basis.<br /> * The current object and one modified version of object are kept (this is different than full bucket versioning). <br /> * The modified version is kept for 6 month - after that period only the current object is kept. <br /> * Deleted objects are kept for 6 months.<br /> <br /> = Backup Procedure =<br /> <br /> Please follow this procedure to request backups of your buckets. <br /> <br /> ==List bucket==<br /> <br /> Send a list of buckets to back up to [mailto:juno@calculquebec.ca sd4h support] with the name and ID of the project where the buckets live.<br /> <br /> ==Give us permission==<br /> <br /> You need to configure the iam policy statement of '''all the buckets''' you want to back up so your TSM robot user in charge of the backup can access them. Here is the policy that needs to be added.<br /> <br /> For example, using the [https://docs.aws.amazon.com/cli/latest/ aws cli], apply the policy on <code>my-bucket</code> using the <code>my-profile</code> identity. <br /> <br /> First, we make sure that <code>my-bucket</code> currently has no policy. <br /> <br /> <pre>$aws s3api --profile my-project get-bucket-policy --bucket my-bucket<br /> <br /> An error occurred (NoSuchBucketPolicy) when calling the GetBucketPolicy operation: The bucket policy does not exist<br /> </pre><br /> <br /> If that command returns something, you need to add the new statement the existing policy. But we are not covering that here.<br /> <br /> Adding policy.json to my-bucket<br /> <div class="filename">'''File :''' policy.json </div><br /> <syntaxhighlight lang=json file=my-policy.json><br /> {<br /> "Statement": [<br /> {<br /> "Effect": "Allow",<br /> "Principal": {"AWS": ["arn:aws:iam:::user/tsm"]},<br /> "Action": [<br /> "s3:ListBucket",<br /> "s3:GetObject"<br /> ],<br /> "Resource": [<br /> "arn:aws:s3:::my-bucket/*",<br /> "arn:aws:s3:::my-bycket"<br /> ]<br /> }<br /> ]<br /> }<br /> </syntaxhighlight><br /> <br /> Then loading the policy to the bucket:<br /> <br /> <pre><br /> $aws s3api --profile my-profile put-bucket-policy --policy file://my-policy.json --bucket my-bucket<br /> </pre><br /> <br /> =Restore Procedure=<br /> <br /> ==List bucket==<br /> <br /> Send us list of buckets or object to restore to [mailto:juno@calculquebec.ca sd4h support]. <br /> <br /> ==Give us permission==<br /> <br /> You will be asked to create a bucket for each bucket you want to restore to restore with the <code>-restore</code> suffix. <br /> <div class="filename">'''File :''' policy.json </div><br /> <syntaxhighlight lang="json" file="my-policy.json"><br /> {<br /> "Statement": [<br /> {<br /> "Effect": "Allow",<br /> "Principal": {"AWS": ["arn:aws:iam:::user/tsm"]},<br /> "Action": [<br /> "s3:ListBucket",<br /> "s3:GetObject",<br /> "s3:PutObject",<br /> "s3:PutObjectAcl",<br /> "s3:AbortMultipartUpload"<br /> ],<br /> "Resource": [<br /> "arn:aws:s3:::my-bucket-restore/*",<br /> "arn:aws:s3:::my-bycket-restore"<br /> ]<br /> }<br /> ]<br /> }<br /> </syntaxhighlight><br /> <br /> Once it is done we will restore you data to that folder.</div>

Dbrownlee https://wiki.c3g-app.sd4h.ca/index.php?title=Store_and_Share_data&diff=166 Store and Share data 2025-05-20T17:38:49Z

<p>Dbrownlee: Fonts.</p> <hr /> <div>=Intro=<br /> SD4H has a large Object Store. Objects are accessible through a web API via the Rados Gateway (radosgw) service. Both the S3 and swift API standards are supported by the radosgw.<br /> <br /> In an Object Store, an object is the equivalent of a file on a posix file system. The object store gives users a lot of flexibility, but the steps to do simple tasks like sharing and transferring data involve somewhat of a learning cure. We propose a procedure here so this curve is as gentle as possible. Once done, the procedure will be both more secure and more flexible than sharing data on a share HPC platform or on some VM owned by your group. <br /> <br /> = Configuring S3 access =<br /> <br /> # You first need to have your [[OpenStack Client]] installed and configured. <br /> # Then, with the client you generate an e2c/S3 id and secret<br /> <br /> With the client is installed and the RC files downloaded in step 1 you can create the S3 ID and secret. <br /> <br /> <syntaxhighlight lang="bash" line><br /> $ source $HOME/id/myproject-openrc.sh<br /> Please enter your OpenStack Password for project po-test as user poq: <br /> # Use the same password that you used to connect to the [https://juno.calculquebec.ca/ Juno web page].<br /> # you can now create the credentials<br /> $ openstack ec2 credentials create<br /> +------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+<br /> | Field | Value |<br /> +------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+<br /> | access | XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX |<br /> | links | {'self': 'https://juno.calculquebec.ca:5000/v3/users/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/credentials/OS-EC2/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'} |<br /> | project_id | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |<br /> | secret | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |<br /> | trust_id | None |<br /> | user_id | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |<br /> +------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+<br /> </syntaxhighlight><br /> <br /> The important values here are <code>access</code> and <code><code>secret</code>which are the S3 [aws_]access_key_id and [aws_]secret_access_key respectively. AWS stands for Amazon Web Services, they are the creator of the S3 API.<br /> <br /> <br /> = Manage you S3 buckets with Globus =<br /> <br /> See the [[Globus]] documentation<br /> <br /> = Use a S3 client to manage your bucket =<br /> There are a few clients that can be used to access Ceph S3 api. We recomend [[rclone]], it is fast and and flexible.</div>

Dbrownlee https://wiki.c3g-app.sd4h.ca/index.php?title=Store_and_Share_data&diff=165 Store and Share data 2025-05-20T17:37:36Z

<p>Dbrownlee: Fonts.</p> <hr /> <div>=Intro=<br /> SD4H has a large Object Store. Objects are accessible through a web API via the Rados Gateway (radosgw) service. Both the S3 and swift API standards are supported by the radosgw.<br /> <br /> In an Object Store, an object is the equivalent of a file on a posix file system. The object store gives users a lot of flexibility, but the steps to do simple tasks like sharing and transferring data involve somewhat of a learning cure. We propose a procedure here so this curve is as gentle as possible. Once done, the procedure will be both more secure and more flexible than sharing data on a share HPC platform or on some VM owned by your group. <br /> <br /> = Configuring S3 access =<br /> <br /> # You first need to have your [[OpenStack Client]] installed and configured. <br /> # Then, with the client you generate an e2c/S3 id and secret<br /> <br /> With the client is installed and the RC files downloaded in step 1 you can create the S3 ID and secret. <br /> <br /> <syntaxhighlight lang="bash" line><br /> $ source $HOME/id/myproject-openrc.sh<br /> Please enter your OpenStack Password for project po-test as user poq: <br /> # Use the same password that you used to connect to the [https://juno.calculquebec.ca/ Juno web page].<br /> # you can now create the credentials<br /> $ openstack ec2 credentials create<br /> +------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+<br /> | Field | Value |<br /> +------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+<br /> | access | XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX |<br /> | links | {'self': 'https://juno.calculquebec.ca:5000/v3/users/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/credentials/OS-EC2/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'} |<br /> | project_id | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |<br /> | secret | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |<br /> | trust_id | None |<br /> | user_id | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |<br /> +------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+<br /> </syntaxhighlight><br /> <br /> The important values here are <code>access</code> and <code><code>secret</code>which are the S3 [aws_]access_key_id and [aws_]secret_access_key respectively. AWS stands for Amazon Web Services, they are the creator of the S3 API. <br /> <br /> <br /> = Manage you S3 buckets with Globus =<br /> <br /> See the [[Globus]] documentation<br /> <br /> = Use a S3 client to manage your bucket =<br /> There are a few clients that can be used to access Ceph S3 api. We recomend [[rclone]], it is fast and and flexible.</div>

Dbrownlee https://wiki.c3g-app.sd4h.ca/index.php?title=Store_and_Share_data&diff=164 Store and Share data 2025-05-20T17:36:34Z

<p>Dbrownlee: Fonts only.</p> <hr /> <div>=Intro=<br /> SD4H has a large Object Store. Objects are accessible through a web API via the Rados Gateway (radosgw) service. Both the S3 and swift API standards are supported by the radosgw.<br /> <br /> In an Object Store, an object is the equivalent of a file on a posix file system. The object store gives users a lot of flexibility, but the steps to do simple tasks like sharing and transferring data involve somewhat of a learning cure. We propose a procedure here so this curve is as gentle as possible. Once done, the procedure will be both more secure and more flexible than sharing data on a share HPC platform or on some VM owned by your group. <br /> <br /> = Configuring S3 access =<br /> <br /> # You first need to have your [[OpenStack Client]] installed and configured. <br /> # Then, with the client you generate an e2c/S3 id and secret<br /> <br /> With the client is installed and the RC files downloaded in step 1 you can create the S3 ID and secret. <br /> <br /> <syntaxhighlight lang="bash" line><br /> $ source $HOME/id/myproject-openrc.sh<br /> Please enter your OpenStack Password for project po-test as user poq: <br /> # Use the same password that you used to connect to the [https://juno.calculquebec.ca/ Juno web page].<br /> # you can now create the credentials<br /> $ openstack ec2 credentials create<br /> +------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+<br /> | Field | Value |<br /> +------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+<br /> | access | XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX |<br /> | links | {'self': 'https://juno.calculquebec.ca:5000/v3/users/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/credentials/OS-EC2/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'} |<br /> | project_id | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |<br /> | secret | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |<br /> | trust_id | None |<br /> | user_id | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |<br /> +------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+<br /> </syntaxhighlight><br /> <br /> The important values here are <code>access</code> and <code><code>secret</code>which are the S3 [aws_]access_key_id and [aws_]secret_access_key respectively. AWS stands for Amazon Web Services, they are the creator of the S3 API. <br /> <br /> <br /> =Manage you S3 buckets with Globus=<br /> <br /> See the [[Globus]] documentation<br /> <br /> =Use a S3 client to manage your bucket=<br /> There are a few clients that can be used to access Ceph S3 api. We recomend [[rclone]], it is fast and and flexible.</div>

Dbrownlee https://wiki.c3g-app.sd4h.ca/index.php?title=Object_Store_Quick_Start&diff=162 Object Store Quick Start 2025-04-25T14:21:40Z

<p>Dbrownlee: </p> <hr /> <div><br /> This section covers the required steps to get started with SD4H's object store.<br /> <br /> Some operations can be made in the OpenStack GUI, but features are limited.<br /> CLI tools offer more control and will allow programmatic access to object store resources. <br /> <br /> == Prerequisites ==<br /> * Install and configure your [[OpenStack_Client|OpenStack CLI]]<br /> <br /> == API endpoints ==<br /> {| class="wikitable"<br /> |+Object Store API Endpoints<br /> !Object Store API<br /> !Endpoint<br /> |-<br /> |'''Swift'''<br /> |<code><nowiki>https://objets.juno.calculquebec.ca/swift/v1</nowiki></code><br /> |-<br /> |'''S3'''<br /> |<code><nowiki>https://objets.juno.calculquebec.ca</nowiki></code><br /> |}<br /> {| class="wikitable"<br /> |+Project Specific Endpoint Patterns (for public READ buckets)<br /> !Object Store API<br /> !Project endpoint pattern<br /> |-<br /> |'''Swift'''<br /> |<code><endpoint>/'''AUTH_<PROJECT ID>/<CONTAINER>/<OBJECT>'''</code><br /> |-<br /> |'''S3'''<br /> |<code><endpoint>/'''<PROJECT ID>:<CONTAINER>/<OBJECT>'''</code><br /> |}<br /> <br /> == Getting credentials for the object store ==<br /> <br /> === The Swift Api ===<br /> You can get access to the [https://docs.openstack.org/swift/latest/api/object_api_v1_overview.html Object Store swift Api] directly with the same RC file credential that you created for the Openstack client in the prerequisites step, and the official [https://pypi.org/project/python-swiftclient/ Openstack Swift client]. <br /> <br /> Note that while the S3 Api is more feature rich and has better support, some operations can only be done with the Swift Api which is the native OpenStack Object Store Api. For example, to get the Quota of you account:<br /> <br /> <syntaxhighlight ,lang="bash"><br /> $ source $HOME/id/myproject-openrc.sh # created for the OpenStack client<br /> $ swift stat --lh<br /> Account: AUTH_d5f8b8e8e3e2442f81573b2f0951013b<br /> Containers: 11<br /> Objects: 2.0M<br /> Bytes: 1.1P<br /> Quota Bytes: 1.5P<br /> Containers in policy "default-placement": 11<br /> Objects in policy "default-placement": 2.0M<br /> Bytes in policy "default-placement": 1.1P<br /> Objects in policy "default-placement-bytes": 0<br /> Bytes in policy "default-placement-bytes": 0<br /> Meta Quota-Containers: 1000<br /> X-Timestamp: 1745522890.88092<br /> X-Account-Bytes-Used-Actual: 1287786000326656<br /> X-Trans-Id: tx0000058e846920f427dfe-00680a90ca-83214639-default<br /> X-Openstack-Request-Id: tx0000058e846920f427dfe-00680a90ca-83214639-default<br /> Accept-Ranges: bytes<br /> Content-Type: text/plain; charset=utf-8<br /> Server: Ceph Object Gateway (squid)<br /> Connection: close<br /> <br /> </syntaxhighlight><br /> <br /> You see here an account with 11 Containers (Swift's Containers are S3 Buckets) 2 Million objects, and 1.1 PB used out of its 1.5 PB quota.<br /> <br /> === The S3 Api ===<br /> While the Switft API can be accessed with the OpenStack RC file credentials, the S3 object store maintains its own set of credentials.<br /> <br /> To create S3 credentials for a project/user:<syntaxhighlight><br /> openstack ec2 credentials create<br /> <br /> +------------+--------------------------------------------------------------------------------------------------+<br /> | Field | Value |<br /> +------------+--------------------------------------------------------------------------------------------------+<br /> | access | <S3 ACCESS KEY> |<br /> | links | {'self': 'https://juno.calculquebec.ca:5000/v3/users/<USER ID>/credentials/OS-EC2/<ACCESS KEY>'} |<br /> | project_id | <OPENSTACK PROJECT ID> |<br /> | secret | <S3 SECRET KEY> |<br /> | trust_id | None |<br /> | user_id | <USER ID> |<br /> +------------+--------------------------------------------------------------------------------------------------+<br /> </syntaxhighlight>Usage details coming soon!<br /> <br /> === Using the Object Store ===<br /> Consider using [[rclone]] to access and work with the Object Store.<br /> <br /> ==What an Object Store is and isn't==<br /> (from https://github.com/s3fs-fuse/s3fs-fuse?tab=readme-ov-file#limitations)<br /> <br /> Generally, an Object Store cannot offer the same performance or semantics as a local file system. More specifically:<br /> <br /> *random writes or appends to files require rewriting the entire object, optimized with multi-part upload copy<br /> *metadata operations such as listing directories have poor performance due to network latency<br /> *non-AWS providers may have eventual consistency so reads can temporarily yield stale data (AWS offers read-after-write consistency since Dec 2020)<br /> *no atomic renames of files or directories<br /> *no coordination between multiple clients mounting the same bucket<br /> *no hard links<br /> *inotify detects only local modifications, not external ones by other clients or tools</div>

Dbrownlee https://wiki.c3g-app.sd4h.ca/index.php?title=Rclone&diff=154 Rclone 2025-04-24T17:49:42Z

<p>Dbrownlee: </p> <hr /> <div>[https://rclone.org/ Rclone] is a powerful client that can interact with multiple storage backends, it offers a good support for our Ceph version of the S3 api and has good speed transfer out of the box. It can also be used to mount an Object Store as traditional block file storage. <br /> <br /> == Configuration ==<br /> <br /> First [https://rclone.org/downloads/ download rclone] or use the [https://rclone.org/install/#script-installation script installation]. Then [[Store_and_Share_data#Configuring_S3_access|get your S3 <code>id key</code> and <code>secret</code>]] from Open Stack.<br /> <br /> Create the following file: <br /> <br /> ~/.config/rclone/rclone.conf<br /> <syntaxhighlight lang="ini" line="1"><br /> [my-project]<br /> type = s3<br /> provider = Other<br /> env_auth = false<br /> access_key_id = <S3 ID from previous step><br /> secret_access_key = <S3 secret from previous step><br /> endpoint = https://objets.juno.calculquebec.ca<br /> acl = private<br /> </syntaxhighlight><br /> <br /> You can then list current bucket, create a bucket and then copy a file into it,<br /> <br /> <syntaxhighlight lang="bash"><br /> $rclone lsd my-project:<br /> -1 2024-01-19 14:12:34 -1 backups<br /> -1 2024-03-07 14:23:26 -1 my-bucket<br /> $rclone mkdir c3g-prod:test<br /> $rclone lsd my-project:<br /> -1 2024-01-19 14:12:34 -1 backups<br /> -1 2024-03-07 14:23:26 -1 my-bucket<br /> -1 2025-04-15 18:08:32 -1 test<br /> $rclone copy my-file.txt my-project:test<br /> $rclone ls my-project:test/<br /> 12408 my-file.txt<br /> </syntaxhighlight><br /> <br /> == Mounting an Object Store ==<br /> To allow mounting by non-root users, in /etc/fuse.conf, uncomment:<syntaxhighlight lang="bash"><br /> user_allow_other<br /> <br /> </syntaxhighlight>Mount the Object Store in daemon mode with:<syntaxhighlight lang="bash"><br /> rclone mount <rclone config block>:<bucket> /path/to/mount/dir --daemon --daemon-wait 0 --allow-other --read-only<br /> # For example:<br /> #rclone mount c3g-data-repos:ihec_data /mnt/ihec_data_objstr --daemon --daemon-wait 0 --allow-other --read-only<br /> </syntaxhighlight>A service may be used to auto-mount the Object Store on boot with a service file (in /etc/systemd/system/).<syntaxhighlight lang="bash"><br /> # Mount the ihec_data_objstr, even after a restart<br /> [Unit]<br /> Description=My Object Store automount<br /> After=network.target<br /> <br /> [Service]<br /> ExecStart=/usr/bin/rclone mount <rclone config block>:<bucket> /path/to/mount/point/dir --no-modtime --fast-list --transfers 50 --checkers 50 --allow-other --read-only<br /> # For example:<br /> # ExecStart=/usr/bin/rclone mount c3g-data-repos:ihec_data /mnt/ihec_data_objstr --no-modtime --fast-list --transfers 50 --checkers 50 --allow-other --read-only<br /> ExecStop=/usr/bin/fusermount -u /mnt/ihec_data_objstr<br /> Restart=always<br /> SyslogIdentifier=ihec_data_objstr<br /> User=ihec<br /> Group=ihec<br /> Environment=RCLONE_CONFIG=/home/ihec/.config/rclone/rclone.conf<br /> TimeoutStopSec=30<br /> <br /> [Install]<br /> WantedBy=multi-user.target<br /> </syntaxhighlight>enable and start the service.<br /> <br /> == Mounting a public Object Store without using credentials ==<br /> Public Object Stores may be accessed or mounted as read-only without the use of Open Stack credentials. This relies on a bucket syntax prepended with the Open Stack project ID.<br /> <br /> Your ~/.config/rclone/rclone.conf need not contain an access_key_id and secret_access_key but only:<syntaxhighlight lang="ini" line="1"><br /> [my-public-project]<br /> type = s3<br /> provider = Other<br /> env_auth = false<br /> endpoint = https://objets.juno.calculquebec.ca<br /> <br /> </syntaxhighlight>Then combine the OS project ID and the bucket name like so:<syntaxhighlight lang="bash"><br /> rclone lsd my-public-project:<OS project ID>:<bucket name><br /> # For example:<br /> # rclone lsd my-public-project:d5f8b8e8e3e2442f81573b2f0951013b:ihec_data<br /> # or<br /> # rclone mount my-public-project:d5f8b8e8e3e2442f81573b2f0951013b:ihec_data /mnt/ihec_data_objstr --daemon --daemon-wait 0 --allow-other --read-only<br /> </syntaxhighlight><br /> <br /> == No problems, only solutions ==<br /> <br /> 1. I cannot upload file larger than 48GB. <br /> In some situation rclone is not able to guess the size of the file to upload and use the default value of`--s3-chunk-size 5M` to spit and upload file to the bucket. But since the server has a 10,000 chunk limit, the upload crashes. You can solve that by setting a larger value:<br /> <br /> <syntaxhighlight lang="bash"><br /> $rclone copy --s3-chunk-size 50M my-large-file.cram my-project:test<br /> </syntaxhighlight><br /> <br /> Note that you need the ram of your computer to be larger that chunks.</div>

Dbrownlee https://wiki.c3g-app.sd4h.ca/index.php?title=Rclone&diff=153 Rclone 2025-04-24T17:46:56Z

<p>Dbrownlee: </p> <hr /> <div>[https://rclone.org/ Rclone] is a powerful client that can interact with multiple storage backends, it offers a good support for our Ceph version of the S3 api and has good speed transfer out of the box. It can also be used to mount an Object Store as traditional block file storage. <br /> <br /> == Configuration ==<br /> <br /> First [https://rclone.org/downloads/ download rclone] or use the [https://rclone.org/install/#script-installation script installation]. Then [[Store_and_Share_data#Configuring_S3_access|get your S3 <code>id key</code> and <code>secret</code>]] from Open Stack.<br /> <br /> Create the following file: <br /> <br /> ~/.config/rclone/rclone.conf<br /> <syntaxhighlight lang="ini" line="1"><br /> [my-project]<br /> type = s3<br /> provider = Other<br /> env_auth = false<br /> access_key_id = <S3 ID from previous step><br /> secret_access_key = <S3 secret from previous step><br /> endpoint = https://objets.juno.calculquebec.ca<br /> acl = private<br /> </syntaxhighlight><br /> <br /> You can then list current bucket, create a bucket and then copy a file into it,<br /> <br /> <syntaxhighlight lang="bash"><br /> $rclone lsd my-project:<br /> -1 2024-01-19 14:12:34 -1 backups<br /> -1 2024-03-07 14:23:26 -1 my-bucket<br /> $rclone mkdir c3g-prod:test<br /> $rclone lsd my-project:<br /> -1 2024-01-19 14:12:34 -1 backups<br /> -1 2024-03-07 14:23:26 -1 my-bucket<br /> -1 2025-04-15 18:08:32 -1 test<br /> $rclone copy my-file.txt my-project:test<br /> $rclone ls my-project:test/<br /> 12408 my-file.txt<br /> </syntaxhighlight><br /> <br /> == Mounting an Object Store ==<br /> To allow mounting by non-root users, in /etc/fuse.conf, uncomment:<syntaxhighlight lang="bash"><br /> user_allow_other<br /> <br /> </syntaxhighlight>Mount the Object Store in daemon mode with:<syntaxhighlight lang="bash"><br /> rclone mount <rclone config block>:<bucket> /path/to/mount/dir --daemon --daemon-wait 0 --allow-other --read-only<br /> # For example:<br /> #rclone mount c3g-data-repos:ihec_data /mnt/ihec_data_objstr --daemon --daemon-wait 0 --allow-other --read-only<br /> </syntaxhighlight>A service may be used to auto-mount the Object Store on boot with a service file (in /etc/systemd/system/).<syntaxhighlight lang="bash"><br /> # Mount the ihec_data_objstr, even after a restart<br /> [Unit]<br /> Description=My Object Store automount<br /> After=network.target<br /> <br /> [Service]<br /> ExecStart=/usr/bin/rclone mount <rclone config block>:<bucket> /path/to/mount/point/dir --no-modtime --fast-list --transfers 50 --checkers 50 --allow-other --read-only<br /> # For example:<br /> # ExecStart=/usr/bin/rclone mount c3g-data-repos:ihec_data /mnt/ihec_data_objstr --no-modtime --fast-list --transfers 50 --checkers 50 --allow-other --read-only<br /> ExecStop=/usr/bin/fusermount -u /mnt/ihec_data_objstr<br /> Restart=always<br /> SyslogIdentifier=ihec_data_objstr<br /> User=ihec<br /> Group=ihec<br /> Environment=RCLONE_CONFIG=/home/ihec/.config/rclone/rclone.conf<br /> TimeoutStopSec=30<br /> <br /> [Install]<br /> WantedBy=multi-user.target<br /> </syntaxhighlight><br /> <br /> == Mounting a public Object Store without using credentials ==<br /> Public Object Stores may be accessed or mounted as read-only without the use of Open Stack credentials. This relies on a bucket syntax prepended with the Open Stack project ID.<br /> <br /> Your ~/.config/rclone/rclone.conf need not contain an access_key_id and secret_access_key but only:<syntaxhighlight lang="ini" line="1"><br /> [my-public-project]<br /> type = s3<br /> provider = Other<br /> env_auth = false<br /> endpoint = https://objets.juno.calculquebec.ca<br /> <br /> </syntaxhighlight>Then combine the OS project ID and the bucket name like so:<syntaxhighlight lang="bash"><br /> rclone lsd my-public-project:<OS project ID>:<bucket name><br /> # For example:<br /> # rclone lsd my-public-project:d5f8b8e8e3e2442f81573b2f0951013b:ihec_data<br /> # or<br /> # rclone mount my-public-project:d5f8b8e8e3e2442f81573b2f0951013b:ihec_data /mnt/ihec_data_objstr --daemon --daemon-wait 0 --allow-other --read-only<br /> </syntaxhighlight><br /> <br /> == No problems, only solutions ==<br /> <br /> 1. I cannot upload file larger than 48GB. <br /> In some situation rclone is not able to guess the size of the file to upload and use the default value of`--s3-chunk-size 5M` to spit and upload file to the bucket. But since the server has a 10,000 chunk limit, the upload crashes. You can solve that by setting a larger value:<br /> <br /> <syntaxhighlight lang="bash"><br /> $rclone copy --s3-chunk-size 50M my-large-file.cram my-project:test<br /> </syntaxhighlight><br /> <br /> Note that you need the ram of your computer to be larger that chunks.</div>

Dbrownlee https://wiki.c3g-app.sd4h.ca/index.php?title=Rclone&diff=152 Rclone 2025-04-24T17:45:56Z

<p>Dbrownlee: Added mount (public and private) sections.</p> <hr /> <div>[https://rclone.org/ Rclone] is a powerful client that can interact with multiple storage backends, it offers a good support for our Ceph version of the S3 api and has good speed transfer out of the box. It can also be used to mount an Object Store as traditional block file storage. <br /> <br /> == Configuration ==<br /> <br /> First [https://rclone.org/downloads/ download rclone] or use the [https://rclone.org/install/#script-installation script installation]. Then [[Store_and_Share_data#Configuring_S3_access|get your S3 <code>id key</code> and <code>secret</code>]] from Open Stack.<br /> <br /> Create the following file: <br /> <br /> ~/.config/rclone/rclone.conf<br /> <syntaxhighlight lang="ini" line="1"><br /> [my-project]<br /> type = s3<br /> provider = Other<br /> env_auth = false<br /> access_key_id = <S3 ID from previous step><br /> secret_access_key = <S3 secret from previous step><br /> endpoint = https://objets.juno.calculquebec.ca<br /> acl = private<br /> </syntaxhighlight><br /> <br /> You can then list current bucket, create a bucket and then copy a file into it,<br /> <br /> <syntaxhighlight lang="bash"><br /> $rclone lsd my-project:<br /> -1 2024-01-19 14:12:34 -1 backups<br /> -1 2024-03-07 14:23:26 -1 my-bucket<br /> $rclone mkdir c3g-prod:test<br /> $rclone lsd my-project:<br /> -1 2024-01-19 14:12:34 -1 backups<br /> -1 2024-03-07 14:23:26 -1 my-bucket<br /> -1 2025-04-15 18:08:32 -1 test<br /> $rclone copy my-file.txt my-project:test<br /> $rclone ls my-project:test/<br /> 12408 my-file.txt<br /> </syntaxhighlight><br /> <br /> == Mounting an Object Store ==<br /> To allow mounting by non-root users, in /etc/fuse.conf, uncomment:<syntaxhighlight lang="bash"><br /> user_allow_other<br /> <br /> </syntaxhighlight>Mount the Object Store in daemon mode with:<syntaxhighlight lang="bash"><br /> rclone mount <rclone config block>:<bucket> /path/to/mount/dir --daemon --daemon-wait 0 --allow-other --read-only<br /> # For example:<br /> #rclone mount c3g-data-repos:ihec_data /mnt/ihec_data_objstr --daemon --daemon-wait 0 --allow-other --read-only<br /> </syntaxhighlight>A service may be used to auto-mount the Object Store on boot with a service file (in /etc/systemd/system/).<syntaxhighlight lang="bash"><br /> # Mount the ihec_data_objstr, even after a restart<br /> [Unit]<br /> Description=My Object Store automount<br /> After=network.target<br /> <br /> [Service]<br /> ExecStart=/usr/bin/rclone mount <rclone config block>:<bucket> /path/to/mount/point/dir --no-modtime --fast-list --transfers 50 --checkers 50 --allow-other --read-only<br /> # For example:<br /> # ExecStart=/usr/bin/rclone mount c3g-data-repos:ihec_data /mnt/ihec_data_objstr --no-modtime --fast-list --transfers 50 --checkers 50 --allow-other --read-only<br /> ExecStop=/usr/bin/fusermount -u /mnt/ihec_data_objstr<br /> Restart=always<br /> SyslogIdentifier=ihec_data_objstr<br /> User=ihec<br /> Group=ihec<br /> Environment=RCLONE_CONFIG=/home/ihec/.config/rclone/rclone.conf<br /> TimeoutStopSec=30<br /> <br /> [Install]<br /> WantedBy=multi-user.target<br /> </syntaxhighlight><br /> <br /> == Mounting a public Object Store without using credentials ==<br /> Public Object Stores may be accessed or mounted as read-only without the use of Open Stack credentials. This relies on a bucket syntax that includes the Open Stack project ID.<br /> <br /> Your ~/.config/rclone/rclone.conf need not contain an access_key_id and secret_access_key but only:<syntaxhighlight lang="ini" line="1"><br /> [my-public-project]<br /> type = s3<br /> provider = Other<br /> env_auth = false<br /> endpoint = https://objets.juno.calculquebec.ca<br /> <br /> </syntaxhighlight>Then combine the OS project ID and the bucket name like so:<syntaxhighlight lang="bash"><br /> rclone lsd my-public-project:<OS project ID>:<bucket name><br /> # For example:<br /> # rclone lsd my-public-project:d5f8b8e8e3e2442f81573b2f0951013b:ihec_data<br /> # or<br /> # rclone mount my-public-project:d5f8b8e8e3e2442f81573b2f0951013b:ihec_data /mnt/ihec_data_objstr --daemon --daemon-wait 0 --allow-other --read-only<br /> </syntaxhighlight><br /> <br /> == No problems, only solutions ==<br /> <br /> 1. I cannot upload file larger than 48GB. <br /> In some situation rclone is not able to guess the size of the file to upload and use the default value of`--s3-chunk-size 5M` to spit and upload file to the bucket. But since the server has a 10,000 chunk limit, the upload crashes. You can solve that by setting a larger value:<br /> <br /> <syntaxhighlight lang="bash"><br /> $rclone copy --s3-chunk-size 50M my-large-file.cram my-project:test<br /> </syntaxhighlight><br /> <br /> Note that you need the ram of your computer to be larger that chunks.</div>

Dbrownlee https://wiki.c3g-app.sd4h.ca/index.php?title=Object_Store_Quick_Start&diff=150 Object Store Quick Start 2025-04-24T16:54:48Z

<p>Dbrownlee: Basic description of an Obj Store</p> <hr /> <div><br /> This section covers the required steps to get started with SD4H's object store.<br /> <br /> Some operations can be made in the OpenStack GUI, but features are limited.<br /> CLI tools offer more control and will allow programmatic access to object store resources. <br /> <br /> == Prerequisites ==<br /> * Install and configure your [[OpenStack_Client|OpenStack CLI]]<br /> <br /> == Getting credentials for the object store ==<br /> <br /> The object store maintains its own set of credentials.<br /> <br /> == What an Object Store is and isn't ==<br /> (from https://github.com/s3fs-fuse/s3fs-fuse?tab=readme-ov-file#limitations)<br /> <br /> Generally, an Object Store cannot offer the same performance or semantics as a local file system. More specifically:<br /> <br /> * random writes or appends to files require rewriting the entire object, optimized with multi-part upload copy<br /> * metadata operations such as listing directories have poor performance due to network latency<br /> * non-AWS providers may have eventual consistency so reads can temporarily yield stale data (AWS offers read-after-write consistency since Dec 2020)<br /> * no atomic renames of files or directories<br /> * no coordination between multiple clients mounting the same bucket<br /> * no hard links<br /> * inotify detects only local modifications, not external ones by other clients or tools</div>

Dbrownlee