Skip to content

support ome-ngff on s3 #209

@d-v-b

Description

@d-v-b

It would be great if this library could access ome-ngff data stored on AWS S3, google GCS, etc. Since you have fsspec in your dependencies, I think this would be doable without drastic changes.

  • this logic uses os.path, so you would need to use something that's agnostic to the storage backend. But it's not clear why you even need to check if the path exists -- Zarr will do that for you, once you choose a store class.
  • you are using DirectoryStore. There are two issues with this: first, DirectoryStore is not recommended for anything because it puts all the chunks in a single directory and, with too many chunks, that directory becomes unwieldy. Consider using NestedDirectoryStore as a replacement. Second, if you have fsspec installed along with zarr, then zarr can transparently store stuff on cloud backends with FSStore. FSStore will even automatically parse a string like s3://bucket-name/foo.zarr and pick the correct backend (s3, in this case).

If there's appetite, I could submit a PR (I didn't see a pre-existing one, but maybe I didn't look too closely).

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: ngffOME-NGFF metadata, spec compliance, collectionsarea: zarrZarr integration, storage backends (zarr-python/tensorstore/zarrs)

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions