RFC-9: Zipped OME-Zarr

[jwindhager]

This RFC attempts to synthesize several discussions around single-file (zipped) OME-Zarr storage

Personally, I'm not terribly familiar with other ongoing work or the RFC process, so early feedback and co-authors welcome!

Specifically, feedback is highly wanted/needed regarding scope & terminology ("single-image OME-Zarr", "composite image", ...) and the "yes/no to root folder" discussion.

CC OSSci Zulip chat on zip stores @joshmoore @normanrz @d-v-b @jni
CC 2024 OME-NGFF hackathon team @jluethi @bpavie @leoschwarz @retogerber @perlman
CC Interested people at the OME-NGFF community call @BioinfoTongLI

[github-actions]

Automated Review URLs

• Readthedocs
• render latest/index.bs
• diff latest modified

[normanrz]

Thanks for starting to work on this. It is a very important piece towards better UX for OME-Zarr.
I think this RFC should be more focussed on how the storage of single-image OME-Zarrs should be standardized, with the clear motivation of improving end-user UX (drag/dropability or even double-clickablity)

[normanrz]

I am thinking that maybe this RFC should be unapologetically about conventions for storing OME-Zarr in zip files; and not try to be more general than that. If we achieve that, it will be a major win for UX.
In my mind, the current state is that only a few viewers support zip files (e.g. Neuroglancer) and a few libraries (e.g. zarr-python and tensorstore), but we want more to support it. Also, there is the undecided question whether the zip contains a folder or the entrypoint zarr.json directly. I think this RFC is a great opportunity to foster more support for zip files, by clearly specifying the convention and summarizing the discussion as well as, as a side effect, deciding the entrypoint question.

If we need additional conventions for different stores, in the future, we can write more RFCs.

[jwindhager]

I am thinking that maybe this RFC should be unapologetically about conventions for storing OME-Zarr in zip files; and not try to be more general than that. If we achieve that, it will be a major win for UX.

I concur. However, there was quite some backlash to specifying a storage backend in the past (even for a specialization). I think the skepticism is mostly stemming from the OME-Zarr specs (deliberately?) not defining storage backends/delegating the storage backend definition to the Zarr specs (where the "storage backend topic" seems a bit stalled, at least for zips). When beginning to draft this RFC, I therefore followed your proposal to separate stores from the "single-entrypoint" specs, to not get "hung up" on this topic. Are you now suggesting to revert this separation?

I think it would be helpful to get more people to weigh in to this topic, before investing more time into drafting this RFC.

[normanrz]

I am thinking that maybe this RFC should be unapologetically about conventions for storing OME-Zarr in zip files; and not try to be more general than that. If we achieve that, it will be a major win for UX.

I concur. However, there was quite some backlash to specifying a storage backend in the past (even for a specialization). I think the skepticism is mostly stemming from the OME-Zarr specs (deliberately?) not defining storage backends/delegating the storage backend definition to the Zarr specs (where the "storage backend topic" seems a bit stalled, at least for zips).

I think we can have both. In general, OME-Zarr is store-agnostic. However, for storing OME-Zarr in zip files, we introduce a convention, that enables the UX benefits we intend.

When beginning to draft this RFC, I therefore followed your proposal to separate stores from the "single-entrypoint" specs, to not get "hung up" on this topic. Are you now suggesting to revert this separation?

Maybe I have not been clear in the post, but I did mean to suggest to specify the convention for single-entrypoint images in zip files.

[jwindhager]

I'm currently on vacation and will continue working on this once I'm back - happy to get any feedback in the meantime, obvs!

[mkitti]

I have re-read this a few times, but I still cannot grasp why the title refers to a "single image" nor why the semantics of what constitutes a single image or "composite" is important here. A large multiscale array stills seems like a "single image" to me regardless of origin. Trying to define an image here I think really distracts from the true objective here - reducing the number of keys or files involved.

If the objective is to reduce the number of keys to one so that single file-like usage is available to users of file systems, especially those near acquisition devices - microscopes, I would rather see a specification built upon the current sharding codec. The sharding codec is by design quite permissive, allowing for space within a blob of bytes that may not be array data, or in this context, image data. Notably the shard index can be placed at the end of the file allowing for the beginning of the file to be used for a single file format.

Current two file Zarr layout

In Zarr, with or without the sharding codec, it is currently possible to define a Zarr array with only two keys, or files on a file system:

1. zarr.json
2. a single chunk, e.g. c/0/0/..../0 by default in Zarr v3.

The utility of sharding is only that the single chunk can be broken into multiple inner chunks, which themselves could be shards. The two file layout is already possible without sharding. The sharding codec already gives us a way to index an arbitrary number of internal binary blobs. We should (re)use it.

Packed zarr.json on a single shard

For the rest of this comment, when I refer to "file", I am talking about a single binary blob or object whether it be on a file system or some abstract key-value store.

If the objective is to turn these two files to one file, then we only need some mechanism to pack these files together. If the shard index were at the end of the file, the beginning of the file could be occupied by text equivalent to the content of a zarr.json file perhaps with a small header indicating some magic byte and the length of the JSON content. The end of the file could consist of the shard index as specifified in the sharding codec. If needed the JSON content could refer to other offsets and byte lengths indicating the location of other binary objects within the shard.

Specifically, the the magic bytes at the beginning could be b'{"zarr_format":3,"zarr_json_length":}', using the Python 3 byte literal syntax, such that the header itself is part of the contents that would be a zarr.json in the two file layout.

This single file could be easily convertible to the two file layout above by copying the zarr.json out of the file and using symbolic linking to place the original file at the expected key location c/0/0/.../0. Note that even Windows operating systems support symbolic linking these days. Even without symbolic linking the file could simply be renamed. The point is that array data does not need to be duplicated (e.g. unzipped) for conversion to the two file layout.

If multiple arrays or multiscale data is desired, the location of additional zarr.json or internal shards could be encoded in zarr.json in the header. However, the end of the file should still be a valid Zarr v3 sharding codec index for at least one array.

Using a TIFF header

I propose the above packing hesitantly because I think much of this could be accomplished by reusing existing single file image formats. In particular, TIFF, or perhaps even OME-TIFF, could be employed here instead, achieving immediate compatibility with a large software application base. If needed this TIFF file could be appended by a Zarr v3 shard index, in order to facilitate easy access to the chunks or reuse of the file as a shard. The TIFF format does not disallow extra binary data at the end of the file (or anywhere in the file).

Within a TIFF, the contents of a zarr.json, could be encoded in the ImageDescription tag, where the OME-XML currently lives, or even encoded within a OME-XML file that is present in that tag.

Under TIFF, chunk locations and sizes could be encoded as tile locations and offsets. The GeoTIFF standard allows for 3D "tiles" and multiscale images. Thus the image data would not need to be duplicated. In particular, cloud optimized GeoTIFFs, have already specified how TIFF file metadata can be consolidated at the beginning of a file. Multiple TIFF pages can be employed for additional dimensions with tile offsets pointing within a high dimensional Zarr chunk.

As a proof of concept, I've demonstrated how a single file could be simultaneously a TIFF, HDF5 file, and Zarr v3 shard here:
https://github.com/mkitti/simple_image_formats/blob/main/header_formats.ipynb

Summary

Attempting to define image semantics here is a distraction. Rather, I think we should reuse existing OME or Zarr technologies as much as possible. Packing a zarr.json as a header onto what is otherwise a valid Zarr shard facilitates easy implementation by existing Zarr implementations. They do not need additional dependencies such as a library handling a zip storage or other archive backend. They would simply need to parse some partial JSON or detect of the closing } of the root JSON object or a null byte as is the C convention for terminating a string.

Moreover, reusing TIFF, a technology already employed by OME would achieve immediately compatibility with a large number of applications. If the TIFF is structured similar to cloud optimized GeoTIFFs and appended with a Zarr shard index, the file could be easily placed in a traditional Zarr layout as a chunk encoded as a shard.

[normanrz]

Thanks mkitti! As previously stated, I agree that redefining composite or single image here is a distraction. This should be about recommendations to store an OME-Zarr hierarchy in a single file.

Sharding is certainly an option here, but having a single file instead of two files would be a huge advantage for UX purposes.

I would favor a zip store over a tiff-based or shard-encoded solution. Instead of inventing something new, that would reuse the existing store abstraction that Zarr offers and most implementations support.
Also, I want to mention that in the geo community many people are moving away from cloud optimized GeoTIFF to Zarr.

[mkitti]

Also, I want to mention that in the geo community many people are moving away from cloud optimized GeoTIFF to Zarr.

The folks at Element 84 have also noticed the the similarities between TIFFs and Zarr shards. For example, see this Element 84 blog post from May 2025:

https://element84.com/software-engineering/is-zarr-the-new-cog/

Here's an excerpt from the post:

But doesn’t a shard–a file containing multiple chunks–sound familiar? Like maybe, could we use…

…could we use a COG as a shard?!?

It turns out…we can!

Check out the linked notebook, where we show exactly this. To make this work, we do have to modify our COG by writing the Zarr shard index to the end of it. But it is still a valid TIFF and valid COG, even if it is subtly modified. In doing so we can now choose how we want to work with the data: do we want to access it as a Zarr using xarray and other common Zarr tooling? Or do we want to use the shard directly as a COG and work with it using any number of compatible tools and applications? We can choose!

The Python package tifffile now even implements a Zarr interface via a ZarrTIFFStore:
https://github.com/cgohlke/tifffile/blob/master/tifffile/zarr.py

In the OME context, I think Zarr-enhanced OME-TIFF files make a lot of sense for the need described here and as a transitional technology.

Is there any image viewer that would recognize zip archive as being potentially an image at the moment?

[mkitti]

I think I'm still confused what is being proposed in this RFC. At least the store part of the discussion needs to be done at the Zarr level rather than as part of an OME metadata specification it seems. It seems that stores are explicitly not an extension point:

https://zarr-specs.readthedocs.io/en/latest/v3/stores/index.html

Stores are not extension points since they define the mechanism for loading metadata documents such that extensions can be loaded.

Or is this supposed to be implemented as an array storage transformer somehow?

[jwindhager]

Hi again

Thanks for the input/feedback/clarifications so far, @normanrz @jni @mkitti!

@mkitti, the objective for this RFC is to improve OME-Zarr's UX in conventional use cases. As @normanrz wrote, this is largely about storing an OME-Zarr hierarchy (note: not necessarily just a single multiscales - ergo the "composite image" terminology) in a single file. Sharding addresses technical problems related to large numbers of files, but not UX challenges. Like @normanrz, I would prefer to not "invent something new" like packing a zarr.json onto a single shard (new file format?) or specialized TIFF.

As for your last comment, @mkitti:

At least the store part of the discussion needs to be done at the Zarr level rather than as part of an OME metadata specification it seems.

This is partially the reason for why I tried to remain store-agnostic in this RFC. Unfortunately, the ZipStore topic is a bit stalled on the Zarr level, not least because people seem to disagree to what extent implementation details (e.g. store interfaces) should be part of the Zarr specification at all (see the links in the "prior art and references" section of this RFC). IMO, the OME-NGFF community cannot (and doesn't need to) wait for this to be resolved upstream. But even if ZipStores were specified upstream, this likely wouldn't address UX-relevant challenges such as discovery of data within zipped OME-Zarrs or the "single-image" semantics for tools to rely on.

---

Overall, the main points of friction so far appear to be:

1. The need for the proposed specialization in general
2. The introduction of new terminology ("composite image")
3. Trying to remain store-agnostic instead of making this about ZIPs

Regarding the first point, I'm afraid the "background" section in this draft (together with the additional context linked in "prior art and references") is my best rookie shot at succinctly explaining why I think this RFC is necessary. I appreciate any pointers as to what parts remain unclear / require more work! And coauthors welcome of course :)

As for points 2 and 3: These are relatively opinionated topics and tbh I'm a bit unsure how to address them. For now, unless anyone has a better idea, I'll just go ahead and see whether I manage to (point 2) rephrase this RFC to rely less on the "composite image" definition and (point 3) strike some middle ground by defining ZIP as RECOMMENDED storage backend.

Happy to get further input in the meantime, obvs!

[mkitti]

Let me try to explain in another way.

An image file, or any binary file, is typically identified through two mechanisms

1. A file extension
2. A set of unique magic bytes, usually at the beginning, of the file

These two items enable the single file UX that is your objective.

A major issue with recognizing Zarr in applications is that there is no magic bytes sequence that can be quickly recognized. As proposed, a consolidated Zarr archive can only be recognized after extracting and parsing the zarr.json.

https://en.wikipedia.org/wiki/List_of_file_signatures?wprov=sfla1

For example, we may want to consider how the command line utility file would recognize Zare archives as such.

A comparable UX experience would be viewing HDF5 files using VS Code via h5wasm, a web browser using h5web, or HDF Viewer.

Here are some examples:
https://h5web.panosc.eu/h5grove?file=epics.h5
https://myhdf5.hdfgroup.org/view?url=https%3A%2F%2Fzenodo.org%2Frecord%2F6497438%2Ffiles%2Fxrr_dataset.h5%3Fdownload%3D1

Note the HDF5 experience has no restrictions on a single image archive.

[jwindhager]

Thanks for the explanation, @mkitti. Tbh, I'm not sure I understand how it relates to your previous comments, but it definitely raises the important point of file type detection protocols (file extensions, magic bytes, MIME headers, ...). As part of the work leading up to this RFC, we briefly touched upon this in zarr-developers/zarr-specs#311 (comment) (also linked in the "prior art and references" section).

Specifying file type detection protocols would certainly improve UX, but they - like resource identifier schemes - tend to become a point of friction in the specification process and aren't currently specified in the OME-Zarr or Zarr specs either. Since this RFC in its current form already sparks quite a bit of discussion (see my attempt at a summary above), I felt that it may not be the right place to introduce even more potentially controversial topics and instead save them for future RFCs. But maybe I'm being too careful here... In any case, I shall at least include those in the "future possibilities" section of this draft!

[mkitti]

Of the six recommendations, five of them are pretty easily verified to be used or not.

The hardest to verify is the fourth recommendation.

4. The root-level zarr.json file SHOULD be the first ZIP file entry and the first entry in the central directory header; other zarr.json files SHOULD follow immediately afterwards, in breadth-first order.

To verify this, an implementation would need to parse the entire central directory which is exactly what should be avoided in case the number of entries is large.

It may be useful to have something in the comment specifically stating whether implementations can assume that all the zarr.json files come first. If I wanted to list all the arrays and groups within the archive, can I just read the central directory until the first non-zarr.json file to get this information?

{
    "ome": {
        "version": "XX.YY",
        "zip_central_directory_order": {
            "metadata_first": true
        }
    }
}

[jwindhager]

As discussed with @joshmoore (editor), this RFC is now considered ready for review. Further changes may be proposed in separate PRs during the RFC phase.

I re-requested a PR review from @normanrz @mkitti (co-authors) for confirmation.

[joshmoore]

Taking the move to D5 as a clear 👍 from the authors (confirmed separately). Amazing work all in getting this together in ready for the hackathon this week. Thanks so much for the dedication. 👏🏽👏🏽👏🏽

[joshmoore]

I re-requested a PR review from @normanrz @mkitti (co-authors) for confirmation.

Apologies, @jwindhager. We must have been editing at the same time; I missed this. @normanrz and @mkitti please comment if you don't support the merging (and/or feel free to add your review) and I'll handle as appropriate.

[imagesc-bot]

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/rfc-9-zipped-ome-zarr/117630/1