OME-NGFF v0.5 (with Zarr v3) implementation plan

[ziw-liu]

Rough steps towards #257:

• Add dispatch to handle both NGFF v0.4 and v0.5 metadata, mostly in iohub.ngff.nodes. The schema itself stays the same between v0.4 and v0.5 so iohub.ngff.models can largely stay untouched. However note that there are added constraints on the Zarr dimension names (part of Zarr v3 spec) outside of the OME field. The goal is to default to v0.5 for creation of new datasets, but still maintain r/w of v0.4 datasets.
• Update the converter to write v0.5.

[srivarra]

Thoughts on zarrs-python? It provides a faster Zarr I/O pipeline and is (surprisingly?) straightforward to incorporate:

import zarr
import zarrs
zarr.config.set({"codec_pipeline.path": "zarrs.ZarrsCodecPipeline"})

The devs made some benchmarks here.

The following plot shows a benchmark of time and peak memory usage to "round trip" a dataset (potentially chunk-by-chunk) as they say.

It's got a couple of limitations:

1. It's Zarr 3 only
2. So far, it only supports the following subset of stores: LocalStore, HTTPFileSystem.
3. Indexing stores with NumPy arrays falls back to the default zarr-python implementation.

[ziw-liu]

@srivarra I tried in #311. Unfortunately zarr-python itself is broken in a multiprocessing setting so a faster codec won't help.

[srivarra]

Ahh, I see, what a bug, this was certainly an interesting read: zarr-developers/zarr-python#3299