Test openDAL

[drernie]

Can this:

• Create and retrieve sha-256 hashes
• support multipart uploads
• Retrieve (and use) versionID

https://opendal.apache.org/docs/rust/opendal/

[drernie]

See also:

https://opendal.apache.org/docs/services/s3

They discuss encryption, but don't seem to have direct checksum support. Could possibly be added by a layer.

They do have detect_region...

[sir-sigurd]

also does it support parallelism on chunk level? i.e. can it upload 1000 chunks concurrently from multiple files?

[drernie]

Well, it seems that they do support multipart upload from S3. We’d have to test how smart it is, and how it handles concurrency. apache/opendal#4282 /// ## Multipart Uploads Write /// /// OpenDAL abstracts the multipart uploads into [`Writer`]. It will automatically /// handle the multipart uploads for you. You can control the behavior of multipart uploads /// by setting `buffer`, `concurrent` via [`Operator::writer_with`]

…

On Tue, Mar 12, 2024 at 05:46 Sergey Fedoseev ***@***.***> wrote: also does it support parallelism on chunk level? i.e. can it upload 1000 chunks concurrently from multiple files? — Reply to this email directly, view it on GitHub <#92 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAE2TYMPYBZR543ZEH2UWTYX32IRAVCNFSM6AAAAABEQ265Y2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJRGU3TEMRWGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[drernie]

FYI, they haven't even finished implementing versioning yet:

apache/opendal#3943

[drernie]

Versioning has been added. May be worth revisiting...