Canonical way of representing paths to other objects

[clbarnes]

Both RFC5 (ping @jo-mueller @bogovicj) and RFC8 (ping @normanrz) have the feature of referring to other zarr objects; either nodes, or metadata fields. We should probably settle on one representation for these which covers the necessary use cases.

Zarr node reference use cases, from simple to complicated:

1. reference to zarr node (group or array) in the same hierarchy, supporting only relative paths with no upward traversal
2. as above, plus upward traversal

• ⚠️ sub-hierarchies can be mounted/ symlinked, so depending on how a node is accessed, different clients may have different trees to traverse

3. as above, plus absolute paths

• ⚠️ the root node can only be discovered by looking for zarr.json in ancestor directories (which has the same problems as upward traversal above), and finding one missing is not sufficient because intermediate groups with no metadata don't need a zarr.json, so you always have to go to the root of the store (which might mean you hit additional authentication problems)

4. as above, plus references to external stores

• ⚠️ there is no canonical way to represent URLs to external stores; the closest are the conventions used by fsspec/ object_store
• ⚠️ the client would need to maintain a registry of parameters for accessing these external stores, including out-of-band authentication if required
• ⚠️ these are fragile to changes in location, e.g.
  • from a pre-publication local location to a published repository
  • from one institution to another if a research group migrates
  • when AWS kills the Open Data program

This may be best handled at the zarr level (ping @d-v-b), but encoding external store references in the Zarr spec may mean encoding more stores (currently filesystem is the only store in the spec).

Metadata reference use case, from simple to complicated, on top of the levels above:

1. access to arbitrary metadata e.g. in an attributes or extension field

• we can use JSON Pointer for this
  • JSON Path, JMES Path, and jq (or jaq) are alternative languages of increasing complexity for accessing JSON: multiple strings could resolve to the same location, and they allow editing the returned data. JSON Pointer is simply a deterministic reference to a specific location in a JSON document.
• ⚠️ need to determine whether the starting point is the zarr.json root object "", the "/attributes" object, the "/attributes/ome" object etc.. The first presupposes that zarr v3 will never change (out of our control); the last leads to shorter paths and allows access to pre-v0.5 fields.
• ⚠️ JSON Pointer addresses array members only by index, and so is sensitive to changes in order
• ⚠️ deserialised metadata may not have the same shape as the JSON form (e.g. a reader may always parse an array of objects with a name field into a map indexed by name), making arbitrary JSON queries difficult

2. access to context-sensitive OME metadata, e.g. where the field containing the reference expects a coordinate system, it would be nice to shortcut the reference so it only looks at the coordinateSystems field of the referenced OME metadata.

• this also makes it more robust to OME-Zarr spec updates

It may be possible to pack all of this into a single string by abusing the fragment and query portions, but it would probably be more clear and flexible to use an object representation, possibly allowing a string for the simplest case (referring to a zarr node in the same hierarchy without upward traversal i.e. a path starting with ./ and not containing ..).

So, I'd suggest something like an object with fields:

• node: string which MUST be interpreted relative to the store root, if given, or the current node, and possibly should not support upwards traversal (or maybe we just hand the users their own footgun, with a warning label). Omission means the root of the store (if given) or this node if not.
• store: optional URL (something something fsspec conventions) to some zarr node
  • ideally this would be something like a PURL where an admin can update the redirect if the data moves, although PURL only accepts specific schemes so isn't something we could use directly. OME-PURL? 😬
  • the same mechanism used for retrieving different stores for different targets can be used to resolve particular PURLs to local paths pre-publication. N.B. we would probably want to make some recommendations about caching redirects
• pointer: optional JSON Pointer string applied to the zarr attributes object. The presence of the pointer field marks this as a reference to a metadata item.

Possibly, we may also want a type field to distinguish whether the reference is pointing to a "group", "array", or "attribute". Then we could extend the types to e.g. a "coordinateSystem" or "transformation", which would do the shortcutting mentioned above and could have different fields to refer to the object by name instead of index.

If we were to allow a uuid or @id in every OME-Zarr metadata type, then we could have a targetId field here to sanity-check that we got the right one.

A field containing a reference could then either be the object, or for the simple case a string, which is interpreted as {"node": "that_string"}.

This also opens the question of whether we would support references in any field (e.g. an array of coordinateSystems which are a mixture of inlined and referenced systems), not just fields which must be references. To which I say 🤷

[normanrz]

Thanks for bringing this up. It is in scope for RFC-8 to unify how references in OME-Zarr work, while incorporating references that RFC-5 introduced and have been in OME-Zarr already (e.g. multiscale dataset paths).

A few notes:

• It is not sufficient to only reference Zarr nodes. In RFC-8 we are introducing standalone collections, which are just json files and don't have to be linked to Zarr hierarchies. With the extension mechansim introduced in RFC-8, OME-Zarr will be able to reference other data formats as well.
• In my view, the "Store" is one of the weakest abstraction that the Zarr specification has to offer, because they are so implementation-specific. There are not canonical roots for stores, becuase the users/implementations can simply choose where to open stores. I wouldn't recommend piggy-backing on that abstraction.
• I think it is out-of-scope for OME-Zarr to solve the stability of references across the Internet. Redirection services can alleviate that risk to some degree, though.

We had some in-depth discussions last week on how we want to propose references for RFC-8. I'll try to summarize here and will write it up in early December. Examples and example code can be found here: https://github.com/normanrz/ngff-rfc8-collection-examples

• References to objects within the JSON document. Each object that is supposed to be referencable needs to have a unique ID within the JSON document (preferably a globally unique ID), in a new id key, e.g. a coordinate system { "id": "839A70E7-D0AF-4038-978D-C4DB792D7B07", "name": "world", "axes": [...] }. References can then be made in the form of an object with a single ref key.

  { "ref": "839A70E7-D0AF-4038-978D-C4DB792D7B07" }

• References to other Zarr nodes (groups and arrays). References can be made by an object with type: zarr and path keys. The type indicates to the implementation that an zarr.json needs to be appended to the path to actually find the metadata. This would reference the root object under attributes/ome (e.g. for a collection) or a Zarr array (e.g. for a singlescale of a multiscale) depending on the context.

  { "type": "zarr", "path": "./path/to/zarr/array" }

• References to other files (e.g. standalone collections). References can be made by an object with type: file (or json tbd) and path keys.

  { "type": "file", "path": "./path/to/collection.json" }

• References to objects in other documents. References can be made by an object as above with an additional ref key.

  { "type": "file", "path": "./path/to/collection.json", "ref": "839A70E7-D0AF-4038-978D-C4DB792D7B07" }

Notes:

• This approach is inspired by jsonschema, where $id, $ref and $anchor keys are used, which itself is modeled after web documents. I think that has a few advantages over JSON pointer; a) it is independent of the structure (cf. the array-index issue that you described), b) it could be translated to other non-json data formats, and c) also robust to OME-Zarr spec changes.
• We are not using Zarr store semantics, but file path/URL semantics. This is to allow for non-Zarr data formats and is just better specced-out than Zarr stores. Currently, relative (to the JSON document) paths, absolute file system paths and HTTP(S) URLs are allowed in RFC-8. Other paths (e.g. S3) or nested paths (e.g. inside a zip archive) are future work.
• I agree that objects for references are much better than relying on string-parsing. However, sometimes it is useful to pass around a single string for which I want to mention the ZEP 8 draft.

[clbarnes]

Thanks Norman! Apologies for retreading ground I'm sure was discussed at the hackathon, I mainly wanted to raise it here so that there's a trail of breadcrumbs.

I agree that stores are an overly-flexible abstraction, but they're also inescapable if we want to make references addressable and actually resolve external data. Introducing a second storage abstraction where OME-Zarr storage semantics form a Venn diagram with Zarr stores and they both need to be able to refer to each other (OME-Zarr store addresses in zarr attributes; zarr store addresses in OME-Zarr collections) seems like a lot of complexity. Piggybacking off ZEP 8 seems a reasonable way to go long-term (if that ever lands).

Worth noting that JSON Schema's $ref can contain a JSON Pointer in the fragment, as is used when referring to the local $defs object.

[d-v-b]

a few thoughts:

1. A nice property of Zarr data is the ability to recursively copy a group + its members to a new location without breaking the semantics of that group. This is nice for a situation where a user just needs a small piece of a larger data set. This property only holds if all references between nodes are relative and directed from parent nodes to child nodes. If we want to keep this nice property, we might want to require these two properties for references between nodes.

2. Another nice property of Zarr data is the ability to rely on a single storage layer. In other words, it's nice if whatever storage API you used to read the first zarr.json document is also sufficient to do all the IO you need to make sense of the rest of the hierarchy. I think this is a weaker form of property 1. This property only holds if all references are absolute or relative paths on the same storage backend.

3. If we want to keep properties 1 and 2, but also work with references to Zarr data on many different backends (a basic thing for a data visualization tool), then it might be worth having different rules for references in different contexts. For example, a free-floating collections JSON object that defines a viewer state must use URLs for everything, but a collections JSON object defined in zarr.json under the ome key is restricted to relative paths.

4. We might want to think of the particular JSON structure of a metadata document as an implementation detail, and for that reason we might not want to use json query style references to individual JSON objects, because these references will be brittle. For example, ome-zarr 0.4 -> 0.5, all ome-zarr metadata got moved under the ome key. This would break any reference to particular JSON objects, but it would not break references defined at a higher level, like "the first multiscale image".

[clbarnes]

This is a duplicate of #144 and from the sounds of it will pretty much entirely be subsumed by RFC 8. In terms of breaking down work into manageable chunks I think it would be beneficial for references to not have to be tied up with the much larger RFC 8 changes (RFC 8 would then just consume the references spec rather than defining it on top of the many other changes to be made), but probably best to centralise on other issues.