Coalescing ManifestStore.get_many

[TomNicholas]

Implements the chunk-coalescing idea from #947: teach ManifestStore to satisfy a batch of chunk requests with fewer, larger reads by deriving an "effective shard index" from the manifests at read time.

ManifestStore.get_many (overriding the new cross-key Store.get_many) resolves the requested chunk keys to (source file, byte range), groups requests by source file, coalesces each group into runs, and serves each run with one ranged read that is sliced back into per-chunk buffers. No file-format assumptions, no spec changes. Keys that aren't plain manifest-backed chunks (metadata, inlined, missing) fall back to per-key get.

Coalescing uses the object_store / async-tiff two-knob model: references merge into one read when their gap is <= coalesce_max_gap_bytes and the merged read stays <= coalesce_max_bytes.

Why coalesce_max_gap_bytes defaults to 0

Benchmarking a 2D map-tile query (5×5 chunks of air_temperature) against a remote Met Office file on Azure, versus the status quo of one get per chunk:

config	reads	fetched	over-read	median	vs baseline
no coalescing (per-chunk get) — status quo	25	621 KiB	1.00×	273 ms	1.00×
gap=0 (merge only adjacent)	5	621 KiB	1.00×	97 ms	2.8× faster
gap=1 MiB, cap=8 MiB	1	1964 KiB	3.16×	456 ms	1.7× slower
gap=1 MiB, cap=256 KiB	5	621 KiB	1.00×	99 ms	2.8× faster
gap=256 KiB, cap=8 MiB	5	621 KiB	1.00×	96 ms	2.8× faster

A 2D spatial box is contiguous along one axis but strided along the other (row-major chunk layout), so a large gap bridges the row stride and pulls in the chunks between rows — reading ~3× the needed bytes and running slower than no coalescing. Merging only adjacent references is a pure win: zero over-read, ~2.8× faster (collapsing 25 requests into 5). Hence the default gap of 0; coalesce_max_bytes (8 MiB) just bounds the size of any single read. A non-zero gap is available for callers who know their access is contiguous.

Depends on the Store.get_many API and its use in BatchedCodecPipeline in zarr-python (zarr-developers/zarr-python#4112, zarr-developers/zarr-python#4113); until those are released this is a dormant override, tested in isolation.

xref #947

Draft — feedback welcome on the approach and the config surface.

[codecov]

Codecov Report

❌ Patch coverage is 98.80952% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 89.93%. Comparing base (80edcb4) to head (7761438).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
virtualizarr/manifests/store.py	98.80%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1033      +/-   ##
==========================================
- Coverage   90.07%   89.93%   -0.14%     
==========================================
  Files          36       36              
  Lines        2206     2275      +69     
==========================================
+ Hits         1987     2046      +59     
- Misses        219      229      +10     

Files with missing lines	Coverage Δ
virtualizarr/manifests/store.py	92.92% <98.80%> (+2.72%)	⬆️

... and 4 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.