Multiple variables in the same shard file

[mjwillson]

Hello!

With zarr v3 sharding, multiple logical chunks can now be stored in a single file, which is great.

Has anyone thought about supporting multiple variables in a single shard file (assuming the variables' shapes are aligned, at least along any dimensions that are chunked)?

We've found it can be a lot more efficient to read data stored this way from cloud storage, when there's a consistent set of variables that you want to read all (or most) of as a group. Likely for similar reasons to why sharding of logical chunks can help: it reduces the number of separate small files that you need to interact with via cloud storage APIs.

This is possible on an ad-hoc basis, e.g. using xarray's Dataset.to_dataarray to stack multiple variables into a single variable, and then saving to zarr, then undoing it after loading the zarr. But it's fundamentally a storage-level optimization and it would be nice if it could be handled at the storage level in a way that's more transparent to the user.

[d-v-b]

I'm not aware of any proposals to do this, since generally we assume that individual Zarr arrays are read and written independently. I can imagine a storage backend that would pack array chunks together, but you would necessarily lose a degree of freedom in how you chunk those arrays.

This seems like a pretty similar outcome to using a record-like data type for a single array, but that would address this problem at the array level.

[mjwillson]

I can imagine a storage backend that would pack array chunks together,

Interesting, yeah if it's possible to implement in a storage backend maybe that's the way to go. (Is sharding of chunks implemented at the level of a custom storage backend though, or if not was there a reason that might also apply here?)

you would necessarily lose a degree of freedom in how you chunk those arrays.

Yeah certainly they'd have to be chunked the same along any dimensions they share, this optimization would only apply in situations where it makes sense.

This seems like a pretty similar outcome to using a record-like data type for a single array, but that would address this problem at the array level.

Interesting, yeah a record dtype could be another way to handle it, although I'm not sure that xarray supports this currently? Perhaps with xarray support, a dataset stored this way would appear the same in xarray as one with separate variables though, which would be a good outcome IMO.

[d-v-b]

Is sharding of chunks implemented at the level of a custom storage backend though, or if not was there a reason that might also apply here?

sharding is implemented as a codec, and codecs are defined independently for different arrays.