assess use of Rclone

[dkropachev]

Since we are using fork of rclone with number of patches that not merged back to upstream.
We have a problem with upgrading it to the top-most version.
In order to define the way out of it we need to assess:

1. Pros and cons of using rclone
2. Effort needed to merge our changes back to the upstream
3. Effort needed to replace rclone with native storage libraries: aws, gcp and azure.

[tzach]

Note that we are also building native S3 access in the Scylla core
https://github.com/scylladb/scylladb/issues?q=is%3Aopen+is%3Aissue+label%3A%22object+storage%22
which can eventually replace the Manager agent upload function.

[tzach]

@bhalevy FYI

[bhalevy]

Yes, we'll be able to copy the snapshots to S3 directly from the scylla node for backup and restore directly from S3 using load-and-stream.

[dkropachev]

Yes, we'll be able to copy the snapshots to S3 directly from the scylla node for backup and restore directly from S3 using load-and-stream.

Is there any plan on gcp and azure support?

[bhalevy]

Yes, we'll be able to copy the snapshots to S3 directly from the scylla node for backup and restore directly from S3 using load-and-stream.

Is there any plan on gcp and azure support?

Only in the longer term.
I understand they provide s3 compatibility of some sort?
Cc @mykaul @xemul @tchaikov

[mykaul]

gcp - https://cloud.google.com/storage/docs/migrating
azure - I don't think there's a compatible stack (without a proxy in the middle).

[karol-kokoszka]

related issues / reason to replace RClone with cloud SDK
https://github.com/scylladb/scylla-enterprise/issues/3462
https://github.com/scylladb/scylla-enterprise/issues/3424