Add Rclone to images?

[wildintellect]

We should consider adding rclone to images, a very useful tool for moving files between various websites and cloud services.

https://rclone.org/commands/rclone_mount/ has the potential to bypass issues like NASA-IMPACT/science-support#13 allowing users to do temporary fuse mounts.

Current install method, due to the general install script wanting to touch /usr/bin it would need sudo. The below method is user doable but only persistent due to home directory mounting. Configurations are written to home directories and usually survive reboots too.

# download
curl https://downloads.rclone.org/v1.73.1/rclone-v1.73.1-linux-amd64.zip --output rclone.zip

# unzip
unzip rclone.zip

# run the tool
./rclone-v1.73.1-linux-amd64/rclone

Options:

• make a script like the above that puts the binary in .local/bin/ for a user ( I can't seem to get a terminal to pick this up on the path - maybe needs reboot?)
• include it by default in the images
• do nothing

[wildintellect]

Noting that conda-forge does have rclone https://anaconda.org/channels/conda-forge/packages/rclone/overview
conda install rclone

This is probably the method we should use.

@grallewellyn have thoughts?

[grallewellyn]

I agree, I made a ticket on the MAAP side to add Rclone to the images that I will manage
MAAP-Project/Community#1274

[wildintellect]

Reading more there's also a question if we want to support arbitrary fusemounts by users which may require an additional system library.
2026/03/11 17:16:27 CRITICAL: Fatal error: failed to mount FUSE fs: fusermount: exec: "fusermount3": executable file not found in $PATH

At least with rclone, sync should work for moving files between ESA/NASA and other arbitrary platforms.

I'll also note that we want the rclone.conf to be in the users Home Directory so it survives boot. Right now it seems to default /etc/xdg/userconfig/rclone/rclone.conf

[wildintellect]

For now if users want to keep their config, every command requires --config path_to_rclone.conf added.

Example:

rclone config --config rclone.conf
# this makes the config file save to rclone.conf
rclone --config rclone.conf lsf aws:bucket-name
# assuming you called your remote store aws

[wildintellect]

This is in the Pangeo image on MAAP, not sure if it's elsewhere yet. Is a really useful add for data transfers.

[yuvipanda]

You can't do arbitrary fusemounts without running as root, so I don't think that should matter

[wildintellect]

@yuvipanda yes we discovered that also, but fuse mounts turn out to be the least interesting part of this. So we're going ahead and adding it as a general transfer tool that is useful for many users bringing in files from other services. Or in the MAAP case, the recommended way to transfer between NASA/ESA.