Guidance on zip-like stores

[clbarnes]

I'm interested in making fjall-rs/sfa accessible through opendal. It's conceptually similar to a zip file but less complicated - the "file entries" (sections) are basically concatenated, uncompressed, and then there's a footer with their keys along with an offset and length. I may also be interested in zip files themselves.

I'd like to be able to read these remotely, e.g. an SFA file on S3. I think that means I should be working on a Layer rather than an Operator, although that might break some assumptions about getting AccessorInfo from the Layer.

I'd intend initially to make this accessor read-only and store the index in memory, probably on first read (so put it behind an RwLock<Option<...>>).

Does this sound like the right track?

---

Mailing list thread: https://lists.apache.org/thread/4547mv3g2bgp2dv7ykh4zf437skmhwr7

[Xuanwo]

It feels more natural to me to design SFA so that the library itself can accept an OpenDAL operator (rather than a Path). This way, all users of SFA can read directly from S3. Is fjall-rs open to accept a change to sfa or building a sfa-opendal?

[clbarnes]

I've already implemented this (but not PR'd it yet) - it's quite a lot simpler than SFA as an Accessor. My thinking was that it would be nice for people to be able to use it as a drop-in replacement for whatever other store they're using.

My use case is Zarr, a scheme for storing and accessing ND arrays in chunks. The rust implementation already has an opendal adapter. There is interest in a single-file format for portability so I thought it would be better to contribute an SFA accessor and hook into existing systems rather than having to write separate bridges for SFA, zip etc..