Write OME-Zarr 0.5

[ziw-liu]

Enable writing OME-Zarr 0.5 (zarr v3). Tested output with:

• ngff-zarr (reading and metadata validation)
• napari-iohub (viewing)
• neuroglancer (viewing)
• ome-zarr-py (metadata validation) with Ome zarr v0.5 reading and writing ome/ome-zarr-py#413
  • napari-ome-zarr (viewing, reading data with the ome-zarr-py branch above)

Also includes miscellaneous typing improvements.

Breaking changes:

• Removed (dysfunctional) option for version 0.1 of OME-Zarr.
• Added sharding option to array creation methods. This is breaking if all the arguments were supplied as positional at the call site.
• Removed bytes from available input path types.
• Replaced the default identity transforms with scale of 1.0 to avoid the schema bug: Add identity to image coordinateTransformations schema ome/ngff#152. This is breaking if an application relied on the default value to be identity.

Performance

Warning

Writing sharded arrays with zarr-python is extremely slow. Writing a 128x2000x4000 array with sharding takes more than 20 minutes! Using tensorstore or zarrs-python is >100x faster.

Other observations on the performance hit when switching to the sharded arrays with zarr-python 3:

• https://github.com/zarrs/zarr_benchmarks
• Performance regression in V3 zarr-developers/zarr-python#2710
• Improve write performance of shards zarr-developers/zarr-python#2977
• Some quick benchmarking of the sharded format zarr-developers/zarr-python#1338
• Support for zarr 3.0.0+ cgohlke/tifffile#297 (comment)
• Extreme Slowness/Timeouts with Large Dataset (Chunked, Multi-Band Zarr Store) zarr-developers/zarr-python#3085
• incremental writing to a sharded array slower than without sharding. zarr-developers/zarr-python#3014
• Unnecessarily slow writes to uncompressed arrays zarr-developers/zarr-python#2518

Other known issues:

• dask.array.to_zarr does not work with shards: Support for sharding when storing dask arrays to zarr dask/dask#11778

[ziw-liu]

cc @bentaculum

[aliddell]

A question for you, but looks good.

[ziw-liu]

@talonchandler @JoOkuma Here's a daxi2 volume converted so that it has 90x smaller chunks but 10x fewer files:

/hpc/projects/comp.micro/test-iohub/daxi/laser_561.zarr

[mattersoflight]

@ziw-liu thanks for your careful work with this.
If I remember @aliddell's benchmark, acquire-zarr writes sharded zarr arrays faster than tensorstore, but doesn't require many dependencies for build.

@aliddell Is acquire-zarr API a drop in replacement for parts of the code that execute the file i/o?

I haven't had a chance to read the code closely, but I am curious if the zarr layout generation and metadata creation are sufficiently separated from file i/o that we can improve the performance for sharded arrays using acquire-zarr.

[aliddell]

@mattersoflight acquire-zarr is not a drop-in replacement, because it doesn't provide random access writing. If iohub had a "streaming mode," then it could suit there, but if you treat your Zarrs as in-memory arrays, it won't.

[ziw-liu]

Merging into the staging branch (#301). We will handle performance in a future PR.