Per-frame metadata in UAPI

[ziw-liu]

In #132 (comment) (@talonchandler) and #103 (comment) (@JoOkuma) the need for accessing each time stamp in a time lapse image is surfaced. However, implementing a clean API at the FOV level is not very straightforward, so we settled on a scalar (uniform delta) for FOVs.

In #151 a more general feature is requested by @ieivanov to access metadata specific to each frame.

IMO the XY frame is indeed a physically grounded entity (readout from the 2D camera sensor), and accessing metadata such as exact time stamp and hardware settings can be useful. It may reduce confusion if these is a shared API across file formats, but I think it adds too much complexity to let frames have their own type, so I'm imagining BaseFOV methods that will return these information given the coordinates. For example:

BaseFOV.frame_camera_timestamp(T: int, C: int, Z: int) -> datetime.datetime:
    ...

And MMStack readers can have their own base class providing the specific hardware information.

[ziw-liu]

One caveat is that these information is not always available. So they cannot be guaranteed as other #132 attributes are.

[ieivanov]

The ZarrReader provides get_image_plane_metadata and the NDTiffReader provides get_image_metadata which return image plane metadata. They also have methods to return the micro-manager summary metadata, which is metadata for the entire acquisition (multiple FOVs). ClearControlFOV provides metadata which, as I read it, is also per acquisition.

It will be useful if the UAPI specifies how to handle per-image metadata (e.g. image timestamp, state of microscope hardware at this image), per-FOV metadata (e.g. FOV XYZ coordinates, FOV label), and per-dataset metadata (e.g. data dimensions, microscope name, user, sample conditions, etc.)

[talonchandler]

I really like the signature you're proposing @ziw-liu:

BaseFOV.frame_camera_timestamp(T: int, C: int, Z: int) -> datetime.datetime:

Building on this, maybe the per-frame metadata can be accessible via:

BaseFOV.frame_metadata(T: int, C: int, Z: int) -> Dict[str, Any]

the per-FOV metadata via:

BaseFOV.metadata -> Dict[str, Any]

and the per-dataset metadata via:

BaseFOVMapping.metadata -> Dict[str, Any]

[ieivanov]

In my opinion the camera timestamp is no different than other image metadata and should be simply contained in the frame_metadata dictionary.

[ziw-liu]

I think this attribute should be an immutable type instead of a plain dict to avoid confusion as the underlying file is read-only. Either a MappingProxyType or dataclass-like interface should work. Copying might be ok performance-wise for single frames but is less explicit about the read-only status.