-
Notifications
You must be signed in to change notification settings - Fork 12
Labels
area: ngffOME-NGFF metadata, spec compliance, collectionsOME-NGFF metadata, spec compliance, collectionsarea: zarrZarr integration, storage backends (zarr-python/tensorstore/zarrs)Zarr integration, storage backends (zarr-python/tensorstore/zarrs)
Milestone
Description
It would be great if this library could access ome-ngff data stored on AWS S3, google GCS, etc. Since you have fsspec in your dependencies, I think this would be doable without drastic changes.
- this logic uses
os.path, so you would need to use something that's agnostic to the storage backend. But it's not clear why you even need to check if the path exists -- Zarr will do that for you, once you choose astoreclass. - you are using DirectoryStore. There are two issues with this: first,
DirectoryStoreis not recommended for anything because it puts all the chunks in a single directory and, with too many chunks, that directory becomes unwieldy. Consider usingNestedDirectoryStoreas a replacement. Second, if you havefsspecinstalled along with zarr, then zarr can transparently store stuff on cloud backends withFSStore.FSStorewill even automatically parse a string likes3://bucket-name/foo.zarrand pick the correct backend (s3, in this case).
If there's appetite, I could submit a PR (I didn't see a pre-existing one, but maybe I didn't look too closely).
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
area: ngffOME-NGFF metadata, spec compliance, collectionsOME-NGFF metadata, spec compliance, collectionsarea: zarrZarr integration, storage backends (zarr-python/tensorstore/zarrs)Zarr integration, storage backends (zarr-python/tensorstore/zarrs)