Add `UUID` field to `multiscales` - moved from PR

[jo-mueller]

Follow-up issue for #115. The gist:

In order to uniquely identify the images that are being generated,
this change proposes that a UUID should be included with each multiscale.
The UUID can either be generated at creation or perhaps copied from
an existing array if all meta(data) is identical (e.g. a rechunking).

In essence, the metadata would contain this:

"multiscales": [
    {,
        "uuid": "urn:uuid:f81d4fae-7dec-11d0-a765-00a0c91e6bf6",
        "name": "example",
        "datasets": [..]
     }
]

[jo-mueller]

@d-v-b @mkitti @clbarnes I'll continue the discussion from ome/ngff-spec#99 here. I was thinking about it a bit, and I think a viable proposal could be:

Every object under any ome key in any ome-zarr metadata document has its own uuid.

This means:

• All metadata objects (multiscales, well, plate, scene) have their own uuid.
• This could pave the way towards more robust referencing down the road?
• Change something in the metadata --> get a new uuid (?)
• Agnostic of the actual array data - checking whether something has changed in a potentially large array is not viable and therefore out of scope for this.

[d-v-b]

can you first describe the problem, and then introduce UUIDs as a solution to that problem? because I really don't understand this at all

[mkitti]

The UUID can either be generated at creation or perhaps copied from
an existing array if all meta(data) is identical (e.g. a rechunking).

So the UUID is meant to identify that two arrays have the same content? What happens if I modify one array or are they read-only once I assign a UUID?

I also do not understand the need for a UUID.

[jo-mueller]

See above: [...] uniquely identify the images that are being generated.

I could see a number of scenarios where this could come in handy, most of which probably revolve around cataloguing: Collections like the ome-ngff challenge right now store links to S3 buckets and URLs. If, at some point, these stores would move to different locations (for whatever reason), updating the references could lead to breaking references. If ome-zarrs had a uuid, it would be possible to validate afterwards whether the link in your catalogue still points to the correct ome-zarr image.

[mkitti]

Every object under any ome key in any ome-zarr metadata document has its own uuid.

Is it only the top level objects under the ome key or all the descendants objects of the key?

[jo-mueller]

So the UUID is meant to identify that two arrays have the same content?

I would see the uuid strictly working on the level of metadata. Even now, the ome-zarr metadata is entirely agnostic to whether the actual zarr data may or may not have changed under its feet. If we wanted a unique identifier for the array data, a uuid would have to live in the zarr metadata, not in the ome metadata

Is it only the top level objects under the ome key or all the descendants objects of the key?

Top-level. For the multiscales, that would look like this:

{
  "ome": {
    "version": 0.x,
    "multiscales": [
      {
        "uuid": "xxxxxxxxxxxxxxx",
        ....
      }
    ]
  }
}

Multiscales are a bit special because they're technically an array of objects that for all practical purposes usually have length 1. So to be very correct, every object in that array would be required to have its own uuid.

For a well:

{
  "ome": {
    "version": 0.x,
    "well": {
      "uuid": .....
    }
  }
}

For a scene:

{
  "ome": {
    "version": 0.x,
    "scene": {
      "uuid": .....
    }
  }
}

For a plate:

{
  "ome": {
    "version": 0.x,
    "plate": {
      "uuid": .....
    }
  }
}

[d-v-b]

If, at some point, these stores would move to different locations (for whatever reason), updating the references could lead to breaking references.

If I host public data at s3://my_bucket and take it down, moving it instead to s3://my_new_bucket, how exactly does a UUID help you here?

[mkitti]

See above: [...] uniquely identify the images that are being generated.

Yes, but what does the UUID indicate here? Objects with the same UUID indicate what exactly? Does it mean that they are exactly the same in terms of content and thus immutable modulo chunking and compression? Or that they only derive from the same original image? What is the semantic meaning of the UUID?

[jo-mueller]

If I host public data at s3://my_bucket and take it down, moving it instead to s3://my_new_bucket, how exactly does a UUID help you here?

If you keep an internal collection of images like this one and the store moves to a different location, maybe the actual images are even re-uploaded somewhere. If the ome-zarr metadata provides a uuid, you can update the links on your end and be sure that you are referring to the same remote object, even if the name has changed.

Conversely, if the image under a given URL is changed (i.e., it's re-generated, updated, whatever), a validator could flag that an entry in a catalogue you are keeping is now actually pointing to a resource that has changed.

Objects with the same UUID indicate what exactly?

The uuid identifies the metadata document, similar to how they're used in ro-crate, for instance.

Does it mean that they are exactly the same in terms of content?

Yes.

and thus immutable modulo chunking and compression?

As said before, it has to be entirely agnostic to that, just because this would

• have to live on the Zarr side of things
• require some sort of hashing of a Zarr array, which is impractical.

Right now, the ome-zarr metadata could never know about any change to the data it describes. But a catalogue could know about changes to the image metadata.

[d-v-b]

If you keep an internal collection of images like this one and the store moves to a different location, maybe the actual images are even re-uploaded somewhere. If the ome-zarr metadata provides a uuid, you can update the links on your end and be sure that you are referring to the same remote object, even if the name has changed.

so we are thinking about a scenario where zarr data is moved from a.zarr to b.zarr, and additionally the layout of the entire hierarchy may have changed. So the original multiscales metadata object at a.zarr/foo/zarr.json#/ome/multiscales/0 is now at b.zarr/<unknown>/zarr.json#/ome/multiscales[unknown].

If all you know is that a.zarr moved to b.zarr, then you have to traverse every zarr.json key under b.zarr and look for a matching UUID. But if you did this, you had to read every metadata document. And if the UUID just identifies the content of the metadata document, then why not use the content of those metadata documents directly (you had to read them all to check the UUIDs!)

Alternatively, you might want identical metadata documents to have different UUIDs. This would make sense when the metadata describes different zarr arrays. But that kind of requires bringing the zarr array into your definition of identity.