Unbounded memory use: /data and /stats read entire datasets into memory

[mvdbeek]

A single request to /data or /stats can read an entire dataset into server memory: DatasetContent.data() with no selection executes dataset[()] (content.py → get_dataset_slice), and an explicit selection can also cover an arbitrarily large region. data_stats() reads the same data plus builds an isfinite mask and a masked copy, and ResolvedEntityContent.attributes() reads all attribute values. The encoders then materialize the full response bytes in memory as well.

Since the shipped FastAPI/Flask/Tornado utils expose these directly, any h5grove deployment can be driven out of memory by requesting a large dataset (or a large slice of one) — no special crafting needed, one GET is enough.

We hit this in Galaxy (usegalaxy.org and many public instances embed h5grove/h5web for HDF5 datasets, where files are user-supplied and multi-GB datasets are routine).

The size of a fixed-dtype read is exactly computable before reading (shape × selection × itemsize), so a few library-level options would let integrators bound this:

1. A public size-estimation API, e.g. DatasetContent.nbytes(selection) computed from the dataset shape, dtype.itemsize, and the parsed selection — integrators can then reject/redirect oversized requests before any I/O. (Caveat: variable-length dtypes can't be pre-sized; h5py exposes no H5Dvlen_get_buf_size.)
2. An optional max_response_bytes setting in the reference FastAPI/Flask/Tornado utils that uses (1) to return an HTTP 4xx for oversized requests.
3. Streaming responses for the formats where it's exact: C-order slab reads concatenate to the same bytes as a full tobytes(), so bin (which h5web uses for numeric data, with a precomputable Content-Length) and npy (header + slabs) can be streamed with bounded memory; csv can stream row batches. data_stats can be computed incrementally per slab (running min/max/count/sum/sumsq) with exact results for any dataset size. json is the hard one (orjson has no incremental API).

We are implementing (1)-style pre-flight guards plus (3)-style slab streaming downstream in Galaxy (PR link to follow once it is open). Happy to contribute an upstream PR for any of these if you're interested — (1) + (2) are small; (3) is more invasive but keeps h5web fully working on datasets far larger than available RAM.

[loichuder]

Hello @mvdbeek and thanks for the detailed issue!

We would be happy to have (1) and (2) in h5grove. As you said, there are small changes but very useful.

(3) looks more challenging but if there is a way of implementing this without too many hurdles, it can be very interesting as well.