Optimization idea: Files as shards?

[TomNicholas]

Can we optimize reading of archival data containing small chunks by pretending that files are zarr shards?

Maximum read throughput from S3 is achieved for objects above a certain size (maybe a few MBs? This number depends on various implementation details, and so is lower in rust than python), but often archival data has chunks < that size, but files > that size. If we can read e.g. one TIFF/HDF5 file in one request, then read throughput from the virtual store could sidestep the bottleneck of the overhead of reading many tiny chunks.

Whether or not this would actually be faster overall than just fetching chunks individually depends on the density of chunks within our "shard" (which can be visualized with @maxrjones' tool). If they are tightly packed then the single shard request would be fetching only data it needs, and this would be a strict improvement. But if they are too spaced out for any reason (e.g. because the file-shard contains multiple variables, with interleaved chunks, which HDF5 definitely does), you might end up fetching too much data that isn't relevant for that array, and it be less efficient overall.

Sharding is Zarr's implementation of internal chunking, where one shard is stored as one object, but contains many chunks, and also contains the shard index, telling you where exactly inside the shards each chunk is. Zarr v3 implements sharding as a codec, so in theory maybe we could change our parsers to return a codec chain containing the ShardingCodec, with the shard index pointing at the internal TIFF/HDF5 chunks?

Unfortunately I don't think that will work, because where would we put the shard index?

• In native zarr v3 it is part of the object, but in virtual stores we can't change the object. So does it have to go in memory? Does that then mean it needs to get serialized into icechunk separately somehow? It's annoying that sharding is a codec and not part of the zarr spec...
• We can't treat the file itself as the shard either because the ShardingCodec assumes a particular format for the index within the object, which won't match pre-existing archival TIFF/HDF5 files.

This just leaves me thinking that zarr should not have implemented sharding as a codec...

cc @sharkinsspatial @jsignell

[maxrjones]

This just leaves me thinking that zarr should not have implemented sharding as a codec...

😆 🫠 😢

A few thoughts:

1. We could batch reads in other ways than sharding. https://github.com/maxrjones/async-hdf5 does this.
2. VirtualiZarr and/or Icechunk could have a utility that produces a sharding codec on the fly based on the contents of an array.
3. It could be possible to develop an extension that specifies a different location for the shard index. Not sure how good of an idea that is though

[d-v-b]

the sharding_indexed codec is just one of many possible array -> bytes codecs that stores subregions of a chunk in separate contiguous byte ranges. if you have some legacy format that you could model as a shard, but the index is incompatible with the rules for the index defined in the sharding_indexed codec, then you can create your own codec specific to that file format that defines the rules for the index.

[jsignell]

I like Max's idea of batching reads in ways other than sharding - I guess you always know which chunks are contained within the same file?

But Davis' idea of a custom codec that implements sharding for an index that is expressed slightly differently is pretty compelling. It seems like this might be better able to understand how the chunks are linearized since it would be customized.

[TomNicholas]

The question is, where do we store the extra information about how the chunks are laid out relative to one another?

1. Nowhere - as @jsignell said technically Icechunk knows which chunks in a manifest come from the same files, and how proximate the byte ranges are - maybe it should just decide how to coalesce requests at read-time?
2. In some format-specific sharding codec - but where do we store the index then? In the codec parameters?
3. In some more general sharding codec that isn't format-specific - It would be nice not to have to have one sharding codec per filetype...
4. In zarr metadata but not as a codec - We currently hack sharding info into the zarr metadata by treating it as a codec, but why isn't sharding information a first-class citizen in the metadata? I'm curious what the rationale was here.
5. In the manifests some other way - Theoretically Icechunk could serialize information about chunk layout at write time, and utilize it at read time, by storing it in the manifest in some other way.

[jsignell]

1 is pretty compelling! That would work for existing virtual icechunk without needing to rewrite the icechunk files.

This might be too far gone, but once there is a Zarr parser and an Icechunk parser (I didn't check those might already exist!) you don't really need the sharding codec at read time as long as you have wrapped the data in a virtual Zarr store.

[d-v-b]

In zarr metadata but not as a codec - We currently hack sharding info into the zarr metadata by treating it as a codec, but why isn't sharding information a first-class citizen in the metadata? I'm curious what the rationale was here.

I wasn't involved in these decisions but I think people saw 2 options: make sharding a storage-level concern, or a chunk encoding concern. Given that sharding involves array indexing (you read different byte ranges depending on which array region was requested), the storage level doesn't look like a good fit, since the storage level is only defined over string keys and byte ranges inside stored objects. it doesn't have a model of array indexing. So they made sharding a chunk encoding ("codec") concern. When you read or write a chunk, array indices are well-defined. so it definitely fits better than at the store level. and sharding does have codec-flavored details, like choosing how to encode the shard index.

[TomNicholas]

1 is pretty compelling! That would work for existing virtual icechunk without needing to rewrite the icechunk files.

Yeah - question is can that be done efficiently...

This might be too far gone, but once there is a Zarr parser and an Icechunk parser (I didn't check those might already exist!) you don't really need the sharding codec at read time as long as you have wrapped the data in a virtual Zarr store.

IIUC you're suggesting ignoring the sharding codec entirely and creating one byte range for every chunk within the shard? The reason not to do that is performance - at small chunk sizes your throughput from object storage would suffer.

I think people saw 2 options: make sharding a storage-level concern, or a chunk encoding concern.

Thanks for the context @d-v-b - that's interesting.

[d-v-b]

to the question "where should we put the index", here is how I am seeing it, lmk if this is off-base:

• an index for a file (hereafter, "chunk") has been computed by a parser specific to that chunk, over a large number of chunks.
• so you have 1 array-like data structure per chunk (the index)
• you don't want readers to re-compute that index explicitly. this would be the "format-specific array-bytes codec" idea, where parsing the legacy file is a process analogous to fetching a shard index for the regular sharding codec. instead you want readers to just read the index from some resource dedicated to holding the index.

conceptually it would be simple to put each index in another zarr array, since each index can be modelled as a 2D array of (byte range start, byte range stop) integers. spec and implementation-wise I don't think anything prevents this! because the spec made sharding a codec, codecs can do IO. You can see this in the zarr-python sharding codec implementation, which has lots of methods that can do arbitrary IO (see https://github.com/zarr-developers/zarr-python/blob/0ea15fd7a2d898ffa445aeacbba9ad64707c9117/src/zarr/codecs/sharding.py#L765).

A simple approach:

• make the chunk indices accessible from the same store as the target array, either in 1 array (with rectilinear chunks if the source data is rectilinearly chunked) or a group with a sub-array per chunk. it doesn't really matter as long as each index is easy to address.
• overload the sharding codec metadata to take a new variant for the index_location field, something like index_location: {"name": "reference", "configuration": {"url": "./index", "node_type": "array"}}, which would denote that the shard indices are saved in a 1D array located in an array located at in the sibling node named index.
• overload the sharding codec implementation to handle this new metadata. instead of reading the index from the start / stop of the shard, the index is fetched the way you want.
• bonus points: if this is all happening in memory, you never need to write anything out: you could do {"url": "memory://<random string>/index"} to fetch chunks from a named memory store scoped to the running process. But this requires features in a PR in zarr-python that has been waiting for review since january 🫠

[LDeakin]

spec and implementation-wise I don't think anything prevents this! because the spec made sharding a codec, codecs can do IO. You can see this in the zarr-python sharding codec implementation, which has lots of methods that can do arbitrary IO

A codec is scoped to operating on a single chunk (i.e. file). A codec that could do truly arbitrary I/O does not align with the Zarr model. Zarr needs something like this: zarr-developers/zarr-specs#346

[d-v-b]

totally agree that my idea violates the spirit of the spec, and would not be likely to gain any traction with other implementations. but it doesn't violate the letter of the spec, because we don't define any bounds on the range of side-effects a codec can have. If your application works entirely with data generated in-memory and you are not worried about format portability, then violating the spirit of the spec might be worth it.

[LDeakin]

Yea I suppose. I guess it is similar to the idea of chunk grids having I/O capability, such that rectilinear could have a non-inline representation of the grid. But it would be a real nice to have for codecs to support multi-file I/O in an official capacity.

[d-v-b]

But it would be a real nice to have for codecs to support multi-file I/O in an official capacity.

Absolutely, and I would support changing the language of the spec to make more clear what operations codec should not perform.