V0.6 Scenes support

[will-moore]

This PR adds support for RFC-5 Scenes.
From the README:

"If the path_to_image.zarr contains a Scene, the coordinateTransforms with their input and output
path/names are used to build a "graph" that includes transforms from the child images. The coordinateSystem
at the "top" of the graph is used to display all the images, with all relevant transforms being applied to
each image. If the graph contains multiple "top" coordinateSystems, the one with the most input images
is chosen for display.
Only the first coordinateSystem from each image is read in order to determine the Axes. The Scene graph is
constructed purely from coordinateTransforms.
Supported coordinateTransforms include identity, scale, translation, rotation, affine and sequence
(containing these other transforms)."

The implementation is quite light on validation and only reads what it needs to create the transform graph.
E.g. No coordinateSystems are used to construct the transform graph. We simply match the input and output from each coordinateTransform to link them together. Any coordinateTransform that doesn't have a dataset[0].coordinateTransform as it's ultimate input is ignored.
The pyramid of dataset resolutions is passed as a stack to napari, without reading any coordinateTransforms apart from the first dataset resolution (same as existing behaviour for multiscale images).
If there is more than 1 coordinateTransform in the datasets[0].coordinateTransformations list, this will lead to the image being duplicated in the graph, as the input from each one is assumed to be the corresponding datasets pyramid (as in the Spec).

We combine the series of transforms that link each image to the "top" coordinateSystem by converting each into a napari affine transform, then multiplying them into a single affine transform which is passed to napari with the image metadata. Any mismatch of dimensions in transforms (e.g. size of matrix or length of 'transforms' list etc) is not validated but will instead result in an error when we try to multiply them or when they are passed to napari.

To test, try the scene at https://ome.github.io/ome-ngff-validator/?source=https://livingobjects.ebi.ac.uk/idr/zarr/test-data/v0.6.dev4/idr0050/4995115_output_to_ms.zarr which has a moderately interesting sequence of transforms, ending with a rotation of the whole scene:

$ napari --plugin napari-ome-zarr https://livingobjects.ebi.ac.uk/idr/zarr/test-data/v0.6.dev4/idr0050/4995115_output_to_ms.zarr

After playing with layer blending and colours to emphasise that the alignment is correct:

This was previously will-moore#4 - now opened against ome main branch

[will-moore]

Testing with examples from https://forum.image.sc/t/ngff-weekly-dev-update-thread/110810/98

https://ome.github.io/ome-ngff-validator/?source=https://s3.zih.tu-dresden.de/johamuel:rfc5-transform-sanity-checks/logan_shepp_rotation_30_clockwise.ome.zarr
This looks like it is rotating around the 0, 0 axis with a rotation matrix:

"rotation": [
    [
        0.8660254037844387,
        0.49999999999999994
    ],
    [
        -0.49999999999999994,
        0.8660254037844387
    ]
]

$ napari --plugin napari-ome-zarr https://s3.zih.tu-dresden.de/johamuel:rfc5-transform-sanity-checks/logan_shepp_rotation_30_clockwise.ome.zarr

So I don't see napari behaving as described will-moore#4 (comment) @jo-mueller ?

[will-moore]

https://deploy-preview-68--ome-ngff-validator.netlify.app/?source=https://s3.zih.tu-dresden.de/johamuel:rfc5-transform-sanity-checks/logan_shepp_rotation_30_degree_anticlockwise_around_center.ome.zarr/
also looks good:

[will-moore]

I tried to open the more complex example at https://ome.github.io/ome-ngff-validator/?source=https://radosgw.public.os.wwu.de/s2v/P2A_B6_M2.ome.zarr/ but this failed due to a 403 response instead of a 404 for /.zattrs. This is also the reason that thumbnails aren't visible in validator for this sample:

$ napari --plugin napari-ome-zarr https://radosgw.public.os.wwu.de/s2v/P2A_B6_M2.ome.zarr
/Users/wmoore/Desktop/ZARR/napari-ome-zarr/napari_ome_zarr/_reader.py:28: UserWarning: Failed to open Zarr group: 403, message='Forbidden', url='https://radosgw.public.os.wwu.de/s2v/P2A_B6_M2.ome.zarr/.zattrs'
  warnings.warn(f"Failed to open Zarr group: {e}")

[will-moore]

If I hard-code the plugin to expect zarr v3 with group = zarr.open_group(path, mode="r", zarr_format=3) then the opening fails with:

  File "/Users/wmoore/Desktop/ZARR/napari-ome-zarr/napari_ome_zarr/ome_zarr_reader.py", line 108, in transform_to_affine
    aff = Affine(affine_matrix=matrix)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/napari_py312/lib/python3.12/site-packages/napari/utils/transforms/transforms.py", line 472, in __init__
    self._linear_matrix = embed_in_identity_matrix(linear_matrix, ndim)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/napari_py312/lib/python3.12/site-packages/napari/utils/transforms/transform_utils.py", line 322, in embed_in_identity_matrix
    raise ValueError(
ValueError: Improper transform matrix [[ 0.94046279  0.0168218   0.04962677]
 [ 0.03821444  0.94170915 -0.22648004]
 [ 0.01767558  0.21776032  0.89960071]
 [ 0.          0.          0.        ]]

[jo-mueller]

So I don't see napari behaving as described will-moore#4 (comment) @jo-mueller ?

The 30° clockwise rotation looks alright to me?

https://deploy-preview-68--ome-ngff-validator.netlify.app/?source=https://s3.zih.tu-dresden.de/johamuel:rfc5-transform-sanity-checks/logan_shepp_rotation_30_degree_anticlockwise_around_center.ome.zarr/
also looks good

😃

If I hard-code the plugin to expect zarr v3 with group = zarr.open_group(path, mode="r", zarr_format=3) then the opening fails

Found the error. In the affine matrix, i wrote the transform in homogeneous coordinates ([0, 0, 0, 1] in affine matrix), which the spec doesn't allow.

@will-moore fixed, should work now

[will-moore]

@jo-mueller - not quite there yet!

$ napari --plugin napari-ome-zarr https://radosgw.public.os.wwu.de/s2v/P2A_B6_M2.ome.zarr/
...
  File "/Users/wmoore/Desktop/ZARR/napari-ome-zarr/napari_ome_zarr/ome_zarr_reader.py", line 108, in transform_to_affine
    aff = Affine(affine_matrix=matrix)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/napari_py312/lib/python3.12/site-packages/napari/utils/transforms/transforms.py", line 472, in __init__
    self._linear_matrix = embed_in_identity_matrix(linear_matrix, ndim)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/napari_py312/lib/python3.12/site-packages/napari/utils/transforms/transform_utils.py", line 322, in embed_in_identity_matrix
    raise ValueError(
ValueError: Improper transform matrix [[ 0.92851552  0.04153813 -0.01662163]
 [-0.08142068  0.98562678 -0.04144036]
 [-0.00377072 -0.01038906  0.78542906]
 [ 0.          0.          0.        ]]

[jo-mueller]

@will-moore shoot, there were more than one affines in the transforms 🙈 Now I should have fixed them all

[will-moore]

@jo-mueller This is looking good at https://ome.github.io/ome-ngff-validator/?source=https://radosgw.public.os.wwu.de/s2v/P2A_B6_M2.ome.zarr/ (thumbnails are working, suggesting a change in the 403 response from the server), but now this has stopped working with napari, even with the zarr_format=3 workaround I mentioned above.
Even trying to open the group with vanilla zarr fails. Do you get this?

>>> import zarr
>>> path = "https://radosgw.public.os.wwu.de/s2v/P2A_B6_M2.ome.zarr/"
>>> group = zarr.open_group(path, mode="r", zarr_format=3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/wmoore/opt/anaconda3/envs/napari_py312/lib/python3.12/site-packages/zarr/api/synchronous.py", line 549, in open_group
    sync(
  File "/Users/wmoore/opt/anaconda3/envs/napari_py312/lib/python3.12/site-packages/zarr/core/sync.py", line 159, in sync
    raise return_result
  File "/Users/wmoore/opt/anaconda3/envs/napari_py312/lib/python3.12/site-packages/zarr/core/sync.py", line 119, in _runner
    return await coro
           ^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/napari_py312/lib/python3.12/site-packages/zarr/api/asynchronous.py", line 866, in open_group
    return await AsyncGroup.open(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/napari_py312/lib/python3.12/site-packages/zarr/core/group.py", line 561, in open
    zarr_json_bytes = await (store_path / ZARR_JSON).get()
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/napari_py312/lib/python3.12/site-packages/zarr/storage/_common.py", line 168, in get
    return await self.store.get(self.path, prototype=prototype, byte_range=byte_range)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/napari_py312/lib/python3.12/site-packages/zarr/storage/_fsspec.py", line 289, in get
    value = prototype.buffer.from_bytes(await self.fs._cat_file(path))
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/napari_py312/lib/python3.12/site-packages/fsspec/implementations/http.py", line 247, in _cat_file
    self._raise_not_found_for_status(r, url)
  File "/Users/wmoore/opt/anaconda3/envs/napari_py312/lib/python3.12/site-packages/fsspec/implementations/http.py", line 230, in _raise_not_found_for_status
    response.raise_for_status()
  File "/Users/wmoore/opt/anaconda3/envs/napari_py312/lib/python3.12/site-packages/aiohttp/client_reqrep.py", line 636, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url='https://radosgw.public.os.wwu.de/s2v/P2A_B6_M2.ome.zarr/zarr.json'

[jo-mueller]

@will-moore Do I have write access to your fork? 😨 Oh no, hope I didn't mess it up with my experiments. I think I made a bit of a mess of things will checking out your branch

[jo-mueller]

@will-moore ok, fixed and loading works on my end. There were some trailing commas in the json, which apparently is not a problem for the json decoder used by the ome-ngff-validator, but it is a problem for the python json decoder 🙄

[will-moore]

Fixed-up sample at https://livingobjects.ebi.ac.uk/idr/zarr/test-data/v0.6.dev4/idr0050/4995115_output_to_ms.zarr - see screenshot in description above

[jo-mueller]

Cool stuff @will-moore! How confident are you to merge? Should we add the opening of the above datasets to the tests? Actually, I think this would also be a nicy addition to that sanity testing dataset on image.sc, if you don't mind :)

[will-moore]

@jo-mueller I haven't looked at this code in a long while and I think there's a bunch of assumptions and TODOs that I need to check on.
Also tidy up print statements etc.
Give me a bit of time to take a look at it... thx

[will-moore]

@jo-mueller OK, I've had a bit of a tidy up. Added some notes for users to the README and added to these in the description above to help reviewers etc understand what's in the PR.

It may be that all this logic is replaced in due course by Scene class from ome-zarr-py and/or transforms from https://github.com/clbarnes/transformnd or something else from OME-Zarr 1.0!
But the timescale and various options for this haven't yet been discussed.

[joshmoore]

@will-moore: perhaps beyond this PR, but what's the best next step here with @clbarnes, @kevinyamauchi & co. to move us towards reusing transformnd?

[jo-mueller]

My headcanon would be:

• Finalize Introduce scene class ome-zarr-py#583, which builds the transform graph
• Use that here to retrieve, traverse and squash the graph into something napari can use.

The general framework will likely stay the same of how things are implemented here (I.e. the usage of the spec class) which could at some point be moved upstream into ome-zarr-py. I can flesh that out a bit more on Monday :)

[jo-mueller]

Hey @will-moore cool stuff! Will try locally with a few sample images, in the meantime find above some minor notes from my side. Also note that in my set of sanity checking datasets, I had the affine matrices written out incorrectly, but it's updated now.

[jo-mueller]

Works like a charm! I also tried loading the example with the displacements field, an informative warning message would be good there.

Otherwise I think this is good to go, as far as we can test it atm.

[will-moore]

With 4f9ff95 and the displacements example above, we see:

/Users/wmoore/Desktop/ZARR/napari-ome-zarr/napari_ome_zarr/ome_zarr_reader.py:131: UserWarning: Unsupported transform type: displacements

[will-moore]

@jo-mueller Made a few fixes from your feedback.

[will-moore]

The last commit comes from testing of various samples I created from ome/ome-zarr-py#582 - handling transforms for a single image in-line with the spec.

• Make sure we only take the first dataset.coordinateTransform
• use that transform to get the name of the "intrinsic" coordinateSystem
• Only use the first multiscales.coordinateTransforms item that has "intrinsic" system as it's input. Previously we just collected ALL the multiscales.coordinateTransforms and applied them all!