consider global attributes for versioning and software identification

[tomkralidis]

Does there exist a convention for providing global attributes for versions or software? An initial proposal:

• version:
• generated_by: the version of the software used to generate the version of the zarr as identified in version

Example:

Attributes: (3/3)
    title:        my dataset
    version:      1.1.0
    generated_by: my-cool-zarr-tool (1.2.0)

[maxrjones]

I don't think a convention already exists for this. The closest is likely the license convention - https://github.com/clbarnes/zarr-convention-license.

There was a lot of discussion at the Zarr Summit about a "provenance" convention, which would be similar to your generated by. Similar art from other fields/formats include Croissant metadata and ISO lineage, though those are much more complicated.

I think your initial proposal is good, though I do strongly prefer name-spaced attributes. I'd like to get @ethanrd, @castelao, and @pvanlaake's thoughts on this too since this relates to best practices for encoding information traditionally covered by CF for NetCDF files in Zarr metadata.

[pvanlaake]

The CF Metadata Conventions define an attribute history to record the date and software library, possibly with function name and arguments, that the data in the data set was generated with. This is an optional attribute which has no mandated structure for the information it contains for each processing step. It can be appended to an individual data variable or to the data set at the global level.

Provenance is a whole different beast which is, IMHO, best left outside of the zarr object for reasons of performance. (For instance, a global land cover map may be constructed from thousands of satellite images, leading to megabytes of provenance metadata. If you combine the land cover map with re-analysis weather data over the current climatological normal period, which is composed of thousands of meteorological satellite images and station observations...) A unique identifier with an external provenance look-up service would probably be a more performant solution.

Back to the issue of software versioning info: How will this be used? Is it merely informative for the human reader or do you foresee any automated processing for a specific purpose?

[tomkralidis]

It would be valuable to have generated_by to be able to determine the underlying software and version used at the point of data production (which may be different than a dataset version global attribute per se). This would help with software defect management and to "replay" a data generation / run pinned to a given software version.

[pvanlaake]

I like the concept but the devil is, as always, in the detail. If an individual array in a Zarr store should have this ability, then the same argument as I made for provenance pops up: How far back do you want to go? I can see 1 step quite easily, but going further back opens up a can of worms containing a lot of metadata. So perhaps restrict this to going back a single step (as in: a single source Zarr array)? I would then still include a unique identifier for the Zarr array that can be used to daisy-chain replays further up the ladder.

@maxrjones What is the thinking on using a UUID or some URI to identify Zarr arrays?

[ethanrd]

Hi Tom @tomkralidis - A few things that might be of interest:

• ACDD (v1.3) has a product_version attribute for versioning datasets and products.
• netCDF has a _NCProperties attribute for netCDF implementation to self identify. It is described in the NUG Appendix A. It is used by the netCDF-C library and I recall discussion around it with, I think, some of the other hdf5/nc4 python libraries but not sure if any (or which) might be using it.
• CF has been discussing provenance for the last year plus. Here's a discussion, "CF Roadmap: Kicking off work on Provenance and Lineage theme" (#389), that contains a summary/conclusion in this comment. It was reported during the 2025 CF Workshop last September. (I do not recall where it stands now.)

[tomkralidis]

Thanks @ethanrd, is there any consideration for dataset metadata reference (so as to point to an ISO metadata record from global attributes, for example)? There would be benefits to this approach (handling in a registry/catalogue), but on the other hand would require infrastructure (remote URL access) to use (vs. having this kind of information shipped in a Zarr).

As a result we could have version and generated_by to support, operational workflow, and things like DOI or (optional) pointers to a metadata record for further discovery.