Collections Specification

[DragaDoncila]

What is an image collection?
A collection of images is a semantic grouping of two or more associated ome-ngff images and/or image-labels.

This definition could include

• Images which do not share a physical coordinate space e.g. training dataset of images containing bees
• Images which share a physical coordinate space and whose storage specification must support sufficient metadata to determine this positioning e.g. high-content screening plates and wells
• A hierarchy of image groups of arbitrary depth which may or may not share physical coordinates
• Other things…?

What workflows should it support?
The specification should support implementations being able to traverse the image collection and, where relevant, map the associated metadata to the physical coordinate space for loading these images.

Ideally, the specification should provide sufficient information at each level of a hierarchical grouping to allow for the loading of both the entire collection, and the loading of an arbitrary level of the hierarchy. This can be important when wanting to share/view partial datasets or update only small parts of the entire collection.

Where labels or other related data is provided (e.g. meshes, points…), the specification should support being able to associate any member of the image collection with its associated labels, regardless of the level in the hierarchy.

The OME-NGFF spec is close to supporting this functionality with the HCS specification which allows the positioning of wells into rows and plates. The main drawbacks of this specification are

• It is too specific to be easily used for images which ARE physically associated but are not HCS acquisitions
• It may be difficult to understand for researchers who are not working with HCS images but nevertheless wish to store their collection in OME-NGFF format
• It does not support an arbitrary depth of groupings
• It does not support collections which are not physically associated

What should it be called?

• Dataset - this term is already used in various places so may not be the best choice
• Collection - a general enough term which is currently mostly unused
• Hierarchical definition - there is a case for this specification being a hierarchy of specifications, with each one defining a more tightly bound collection e.g.
  • Bag - associated images with no metadata
  • Stack - associated images which overlap in physical space
  • Panorama - associated images which stitch together in physical space

Ideally, the names used in the base specification would be general enough to support a broad variety of use cases and tailored use cases could be demonstrated using examples in the documentation.

Reference specifications
BDV XML Files
SVG
TrakEM2
Napari Plugin for image-label collections
mobie grid view of many sources

Related
Image.SC discussion on collections
Live notes from latest community call
HCS Specification

What next?
I think we should first decide on whether we want to support arbitrary levels in the hierarchy and whether we want a general spec which we can “inherit” from for more detailed specs, or whether we want one spec to rule them all.

My vote is that we define the most generic collection (a “bag” of images) which works with arbitrary levels of grouping (it’s collections all the way down), and then work to add to it for more complex collections. I will be working on this over the coming week and will post here once I have something working, but of course would love to hear what everyone’s thoughts are on the best way forward.

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/next-call-on-next-gen-bioimaging-data-tools-feb-23/48386/9

[tischi]

@DragaDoncila Thank you very much for the detailed post! It makes a lot of sense and I am looking forward to whatever you come up with! Incidentally we (cc @constantinpape ) were also working on this topic during the past few days.

I also ping @d-v-b

I would like to add a notion and would be curious to hear opinions:

Currently I think that I would prefer storing images within a zarr container without any hierarchy, i.e. just as a flat list. The main reason is simplicity for the reader and writer libraries (the current HCS specifications does not follow this). Anything that imposes a hierarchy would be handled by the collections specification, which I think could be seen as metadata that specifies how to display and layout several images together.

[constantinpape]

Currently I think that I would prefer storing images within a zarr container without any hierarchy, i.e. just as a flat list.

This means you would like to store images as 2d arrays and volumes as 3d arrays, correct?
I created #35 to discuss this.

[tischi]

This means you would like to store images as 2d arrays and volumes as 3d arrays, correct?

I was not gonna enter the 3D vs 5D discussion here, but just wanted to say that I feel that structuring the zarr like this: https://ngff.openmicroscopy.org/latest/#hcs-layout feels overly complex to me.

[constantinpape]

I was not gonna enter the 3D vs 5D discussion here, but just wanted to say that I feel that structuring the zarr like this: https://ngff.openmicroscopy.org/latest/#hcs-layout feels overly complex to me.

Ok, so your point is to rather have a flat hierarchy of images in the zarr container:

image1/
image2/
image3/
image4/
...

and then define the potential hierarchies in the collections metadata (just a mock-up):

{
  "well1": ["image1", "image2"],
  "well2": ["image3", "image4"]
}

[tischi]

Yes, exactly.

The way I see it conceptually is that a multi-well plate is a specific layout of a bag of images and, as such, should be covered by our collections specification, which I would currently see as metadata that exists independent of the way we store the raw image data. What do you think?

[will-moore]

One feature that the current HCS layout gives us is a URL to a specific Well. So I can open a specific Well like: https://hms-dbmi.github.io/vizarr/v0.1?source=https://s3.embassy.ebi.ac.uk/idr/zarr/v0.1/plates/2551.zarr/A/1
Demo movie at https://twitter.com/will_j_moore/status/1322187662762090497

I guess you could try to use a URL ?query or a #fragment to refer to a Well or other subgroup. E.g. path/to/plate.zarr/#A1

[tischi]

a URL to a specific Well

I see that this is cool, but I am afraid that (i) these hierarchies make it harder parse an ome.zarr and (ii) it is not flexible; for example, I guess I cannot produce an single URL to show me all the images that were subjected to the same biological treatment (which may be several wells).

[DragaDoncila]

The way I see it conceptually is that a multi-well plate is a specific layout of a bag of images and, as such, should be covered by our collections specification, which I would currently see as metadata that exists independent of the way we store the raw image data.

I completely agree that the metadata and storage should be independent, because I think this also provides the opportunity to support a wider range of custom metadata. For example this:

{
  "well1": ["image1", "image2"],
  "well2": ["image3", "image4"]
}

could easily be this (for some geographical feature learning model):

{
  "lakes": ["image1", "image2"],
  "mountains": ["image3", "image4"]
}

I guess that's what I was thinking of when I said

tailored use cases could be demonstrated using examples in the documentation.

I like the idea of a flat set of images with the hierarchy determined entirely by the metadata. That certainly seems the easiest way to support an arbitrary level of hierarchy without ending up with a very complex storage structure.

[joshmoore]

#31 (comment) Currently I think that I would prefer storing images within a zarr container without any hierarchy, i.e. just as a flat list.

Is this a MAY or a MUST? And what happens when/if someone does make use of the folder structure available in Zarr/N5/HDF5?

[will-moore]

Re @tischi "flexibility and biological treatment" - I'm wondering if there must be a single 'hierarchy' in the container, e.g. If we can have multiple. E.g.

{
  "well1": ["image1", "image2"],
  "well2": ["image3", "image4"]
}

And:

{
  "aquisition1": ["image1", "image3"],
  "aquisition2": ["image2", "image4"]
}

or

{
  "drug1": ["image1", "image3"],
  "drug2": ["image2", "image4"]
}

Those are all different ways to grouping the images.
But if you have:

{
  "well1": ["image1", "image2"],
  "well2": ["image3", "image4"],
  "drug1": ["image1", "image3"],
  "drug2": ["image2", "image4"]
}
how do you know which groups are mutually exclusive. E.g. which ones are Wells vv Treatments?

Having multiple hierarchies might provide more flexibility, but this makes it harder to understand how to view the data.
Instead, it might make more sense to only have a single hierarchy (like a file-system) and then add other metadata in other ways?

[tischi]

@will-moore

My current idea would be to have no hierarchy on the data storage level, but provide the possibility to specify different "views" on the data on the metadata level. Something along the lines:

views: 
{
   "well_based": {...},
   "treatment_based":{...}
}
default_view: "well_based"

Does that make sense to you?

[tischi]

Is this a MAY or a MUST? And what happens when/if someone does make use of the folder structure available in Zarr/N5/HDF5?

Personally, I'd be for a MUST, i.e. not support hierarchies and then ignore anything stored at deeper levels.
But, obviously, that's just my personal opinion. Very curious to hear other opinions!