More efficient way to encode "fetch this whole chunk"

[TomNicholas]

Currently if we create a virtual reference that points at an entire object it is stored internally as

(path=s3://bucket/file.whatever, offset=0, length=<full_length_of_object>)

where the <full_length_of_object> needs to be determined at parsing-time.

This leads to a big inefficiency in the ZarrParser, which currently does an O(n_chunks) iteration over all chunks in the Zarr array to discover the sizes of every chunk object. It would be great to be able to skip this iteration.

VirtualiZarr/virtualizarr/parsers/zarr.py

Line 81 in c67dcc1

size = await zarr_array.store.getsize(scalar_key)

It is apparently possible to issue a HTTP range request for a whole object without specifying the length of the object (see allowed range header syntax). So if we instead e.g. stored the virtual reference as

(path=s3://bucket/file.whatever, offset=0, length=-1)

and our chunk-fetching IO implementations (i.e. ManifestStore/Icechunk/fsspec) knew what to do with this, then we could skip getting the object size, and see an O(n_chunks) speedup when virtualizing any (un-sharded) Zarr store.

[maxrjones]

The Icechunk and ObjectStore implementations already handle key requests, offset requests, suffix requests, and range requests from Zarr Python - https://github.com/zarr-developers/zarr-python/blob/1c05c1a10b68ce990374083e8db989357372dff9/src/zarr/abc/store.py#L197-L201 so this shouldn't be a big lift in the store classes.

I would rather us use dataclasses like Zarr-Python rather than HTTPS syntax for the in-memory virtual references (e.g., `-1 for through the end of the object1) because it's more readable with no noticeable performance hit.

But for this to be useful, it'd definitely need to be added to the Icechunk/Kerchunk spec + implementations.

[TomNicholas]

Thanks @maxrjones.

The Icechunk and ObjectStore implementations already handle key requests, offset requests, suffix requests, and range requests from Zarr Python

Excellent.

I would rather us use dataclasses

Sure, we can store in memory in virtualizarr however it makes sense. Though I'm not quite sure how you're proposing to use a numpy array of dataclasses...

But for this to be useful, it'd definitely need to be added to the Icechunk/Kerchunk spec + implementations.

@paraseba / @mdurant do you have any thoughts on this?

[paraseba]

I think this is doable in Icechunk. To do it without requiring a format change, we would have to use 0 length as the marker of "to the end", which is something Icechunk could handle internally. That works unless there is some other usage for 0-length chunks in VZ, but I can't think how that would be useful.

But there are some concerns, we would have to think about it a bit more, because this would be the first type of chunk for which Icechunk doesn't know the size. For example, @TomNicholas recently implemented virtual chunk statistics in Icechunk. This change would render that invalid.