Implement Lazy Loading of HDF5 Assets

[genematx]

Summary

Follow-up on #1463.

#1463 made many-file HDF5 reads reuse a cached Dask graph, but the first read of a dataset in a fresh process still opened every constituent file once to discover specs before the cache could help. This PR extends the lazy asset-resolution path (introduced for file sequences in #1463) to multi-file HDF5 array datasets, so the first read opens only the files its slice actually touches.

Why HDF5 needed its own path

A file sequence stacks uniform frames — one file adds exactly one row on a new leading axis — so the catalog's 1 file = 1 row geometry recovers file boundaries from the structure alone. HDF5 files instead concatenate along the array's leading axis, and each file may contribute a different number of rows, so the boundaries can not be read off structure.chunks.

What changed

• catalog/adapter.py — _get_lazy_adapter is now adapter-agnostic. It no longer gates on a chunks property; it derives the file count from a single aggregate asset query (which also performs the existing num contiguity check), then asks the adapter class file_indices_for_slice(structure, n_files, slice, properties=...) which files a slice touches. Only the adapter interprets properties. It also forwards the data source parameters (as the eager path does) so adapters that need them (e.g. HDF5's dataset) are configured. This additionally extends lazy loading to plain (non-chunks) file stacks.

• adapters/hdf5.py — HDF5ArrayAdapter opts in via supports_lazy_assets and gains _file_layout, which resolves each file's extent along the leading axis in priority order:

  1. an optional per-asset extents property — the authoritative per-file row counts (sum(extents) == shape[0]); supports genuinely non-uniform files,
  2. else structure.chunks[0] when it has exactly one chunk per file,
  3. else the grid geometry (a uniform stack whose leading axis was reshaped across several structure dimensions),
  4. else None → the catalog falls back to the full build.

  _lazy_stack_from_structure builds the file-stacked Dask array from the known structure without opening any file for specs: touched files get a whole-file read task (URI resolved to a path lazily, inside the task), untouched files get a placeholder block that Dask culls once the read slices the reshaped array (and that raises loudly if ever computed, so a geometry mismatch can never serve wrong data).

• adapters/utils.py — the file-alignment geometry shared by the sequence and HDF5 adapters is extracted into grid_shape_for_files (a pure function of shape + file count, no I/O).

On the optional extents property

extents is optional and fully adapter-contained; the catalog stays ignorant of it. Both datasets benchmarked below resolve without it — enc1 via one leading chunk per file (path 2), fluor via the grid geometry (path 3). It is needed only for genuinely non-uniform flat files (differing per-file row counts that are not one-chunk-per-file). A writer can populate properties["extents"] to unlock the lazy path there; that consolidator change is a planned follow-up (not in this PR).

Results

Cold first read (fresh process), arr[0:1]

Measured on a staging catalog by patching h5py.File to count opens:

dataset	before (#1463)	after (this PR)
enc1 (3 files, (3, 11))	900 ms / 8 opens	352 ms / 2 opens
enc1 (153 files, (153, 223))	27.7 s / 282 opens	175 ms / 2 opens
fluor (153 files, (153, 223, 8, 4096) f8)	29.7 s / 273 opens	1.0 s / 2 opens

h5py opens are the deterministic metric: before scales with the file count (every file opened for specs, ~2× for external links); after is a constant 2, independent of how many files back the dataset. Absolute times are filesystem-latency dependent (storage was heavily loaded during the before run), but the open-count reduction is structural.

Value equivalence (lazy vs. eager full build)

The lazy read was compared byte-for-byte against a forced eager full-adapter build across first / middle / last / arbitrary-window / leading-plus-trailing / deep multi-axis slices. All identical:

dataset (path)	slices checked	result
fluor (grid)	[0:1], [76:78], [152:153], [75:77], [0:2, 0:5], [10:11, 100:110, 3:6]	6/6 PASS, max abs diff 0.0
enc1 (chunks extents)	[0:1], [100:103], [152:153], [50:52, 10:20]	4/4 PASS, max abs diff 0.0

Tests

• pytest tests/test_catalog.py tests/test_hdf5.py tests/test_tiff.py tests/test_slicer.py → 263 passed, 29 skipped.
• New unit tests in test_catalog.py cover the grid path (opens only touched files), multi-chunk files, the non-file-aligned fallback, the extents property (parameterized, incl. non-uniform), and the "non-uniform requires extents" case.
• pre-commit (flake8 / isort / black / mypy) all pass.

Checklist

• Add a Changelog entry
• Add the ticket number which this PR closes to the comment section

[danielballan]

extents is optional and fully adapter-contained; the catalog stays ignorant of it.
Does this indicate that extents belongs in parameters? I am not sure whether we took a clear position on:

A) Items in properties have semantic meaning to the catalog/server.
B) Items in properties have semantic meaning that generalizes across Adapters.

If (A) then it sounds like extents should be a parameter. If (B) then it could stay a property.

[genematx]

Interesting. I always thought of properties as something not_required for reading the data (e.g. unlike dataset), but which might help (e.g. to serve the portions aligned with chunks).
To me, it looks like extents belong in properties -- they just help to disambiguate (native) chunks.