☂️ Two extension points: `extendStore` and `extendArray`

[manzt]

Umbrella for the plan and stacked PRs that close out #349.

#349 asked for a blessed way to compose stores. Surveying the real use cases, zarrita actually needs two composition points, not one, and the current AsyncReadable<Options> generic is a symptom of only having the wrong half.

This is the plan:

• zarr.extendStore for the transport layer (bytes in, bytes out)
• zarr.extendArray for the data layer (coords in, chunks out)

import * as zarr from "zarrita";

// Transport layer: wrap a store with caching, batching, consolidated metadata.
let store = await zarr.extendStore(
  new zarr.FetchStore("https://example.com/data.zarr"),
  (s) => zarr.withConsolidation(s),
  (s) => zarr.withRangeBatching(s, { cacheSize: 512 }),
);

// Data layer: wrap an array with chunk caching, prefetch, observability.
let arr = await zarr.extendArray(
  await zarr.open(store, { kind: "array" }),
  (a) => withChunkCache(a, { cache: new Map() }),
);

await zarr.get(arr, [null, zarr.slice(0, 10)]);

Both extension points are built from the same factory primitive:

const withChunkCache = zarr.defineArrayMiddleware(
  (array, opts: { cache: Map<string, zarr.Chunk<zarr.DataType>> }) => ({
    async getChunk(coords, options) {
      let key = coords.join(",");
      let hit = opts.cache.get(key);
      if (hit) return hit;
      let chunk = await array.getChunk(coords, options);
      opts.cache.set(key, chunk);
      return chunk;
    },
  }),
);

Why two layers

Every custom store, wrapper, cache, and hack in the zarrita ecosystem falls into one of two buckets.

Transport concerns operate on (key, range) -> Uint8Array and don't care about zarr's logical model:

• Auth, presigning, request transformation. Covered at the call site by the
  custom fetch option from Add custom fetch option to FetchStore #388.
• Status code remapping (e.g. S3 403 -> 404 on private buckets).
• Range batching and coalescing.
• Consolidated metadata short-circuiting.
• Byte caching, e.g. vizarr's lru(store) wrapper.

Data concerns operate on (chunkCoords) -> Chunk<T> and don't care about paths or bytes:

• Chunk caching (Add optional chunk caching to get() function #296, closed). Tried to add cache?: ChunkCache to
  GetOptions because there was no chunk-level extension point.
• Observability, scheduling, prefetch priority, e.g.
  vole-core's wrapArray,
  which bypasses the generic entirely with a bare Proxy over the Array.
• Virtual-format adapters: hdf5-as-virtual-zarr.js, tiff-as-virtual-zarr.js, parquet-as-virtual-zarr.js. These thread RequestInit through layers that never read it.

Before this refactor, zarrita only had a transport extension point, and even that was unofficial (subclass FetchStore or hand-roll an AsyncReadable). The data layer had no extension point at all, so chunk-level concerns were smuggled through AsyncReadable<Options>, a generic that threaded opaque per-call state from the call site all the way down to the store.

Once the data layer has its own extension point, the Options generic has no job. It goes away, and signal (the one thing anyone ever actually threaded) becomes a plain field:

// Before
interface AsyncReadable<Options = unknown> {
  get(key: AbsolutePath, opts?: Options): Promise<Uint8Array | undefined>;
}
await zarr.get(arr, null, { opts: { signal: ctl.signal } });

// After
interface AsyncReadable {
  get(key: AbsolutePath, opts?: { signal?: AbortSignal }): Promise<Uint8Array | undefined>;
}
await zarr.get(arr, null, { signal: ctl.signal });

Stacked PRs

• Add zarr.defineStoreMiddleware and zarr.extendStore middleware system #384 Add composable store middleware system (defineStoreMiddleware, extendStore, withConsolidation, withRangeBatching)
• Drop Options generic and thread signal directly #391 Drop Options generic and thread signal directly
• Add zarr.defineArrayMiddleware and zarr.extendArray #392 Add zarr.defineArrayMiddleware and zarr.extendArray
• Document middleware extension points and tighten signal tests #393 Document middleware extension points and tighten signal tests

What this unblocks

• Add optional chunk caching to get() function #296 Chunk caching becomes a withChunkCache middleware, roughly 15 lines.
• Options not passed when fetching from a zarr with sharding #306 Signal propagation through sharded arrays is hardened by the type change.
• vole-core's wrapArray scheduler migrates to defineArrayMiddleware + extendArray, dropping the bare-Proxy workaround.
• Downstream stores typed as AsyncReadable<RequestInit> (idetik, orkestrator, carbonplan/zarr-layer, keller-mark virtual stores) collapse to plain AsyncReadable.

[keller-mark]

This makes a lot of sense. Just want to note that the virtual format adapters also have non-array-chunk concerns, as they need to return virtualized array/group/consolidated_metadata (eg for the key /my_image/zarr.json, if my_image is actually an OME TIFF, then the transport layer needs to intercept based on the key to generate the OME NGFF zattrs).

(I don't know if this means a third category of attrs-concerns is needed, or if it just means that the virtual format adapters involve both transport+data.)

Further, in some cases, the virtual format adapters will be able to hand-off pre-codec-decoding bytes (transport concern) to Zarrita (providing corresponding codec metadata for the array) if the underlying format uses a zarr-compatible compression mechanism (or if a custom decoder has been registered with zarrita). In other cases, the adapter will do the codec-decoding internally, and then will want to hand-off decoded chunks to Zarrita (data concern).