Skip to content

Commit 37b46f5

Browse files
committed
Add composable store middleware system
Stores in zarrita can now be wrapped with composable middleware using `wrapStore`. Each middleware intercepts store methods via Proxy and can add new methods, with automatic delegation for anything not overridden. Middleware supports a dual API — direct `withX(store, opts)` and curried `withX(opts)` for use with the new `createStore` pipeline. For middleware whose options depend on the store's request options type (e.g., `mergeOptions` in range batching), `wrapStore.generic` uses a higher-kinded type encoding via the `GenericOptions` interface to flow the store's `O` type through to the options at the call site. `withConsolidated` and `tryWithConsolidated` are renamed to `withConsolidation` and `withMaybeConsolidation` for consistency. The old names are re-exported with `@deprecated` JSDoc. `BatchedRangeStore` class is replaced by `withRangeBatching` built on `wrapStore.generic`. ```ts let store = await createStore( new FetchStore("https://..."), withConsolidation({ format: "v3" }), withRangeBatching(), ); store.contents(); // from consolidation store.stats; // from batching ```
1 parent 53eff05 commit 37b46f5

File tree

10 files changed

+868
-474
lines changed

10 files changed

+868
-474
lines changed
Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
---
2+
"zarrita": minor
3+
---
4+
5+
Introduce composable store middleware via `wrapStore`
6+
7+
**New APIs:**
8+
9+
- `wrapStore(factory)` — define a store middleware with dual API (direct + curried) and automatic Proxy delegation. Supports sync and async factories.
10+
- `wrapStore.generic<OptsLambda>()(factory)` — for middleware whose options depend on the store's request options type (e.g., `mergeOptions`). Uses `GenericOptions` interface for higher-kinded type encoding.
11+
- `createStore(store, ...middleware)` — compose middleware in a pipeline. Returns `Promise` to support async middleware like `withConsolidation`.
12+
13+
**Renamed:**
14+
15+
- `withConsolidated``withConsolidation`
16+
- `tryWithConsolidated``withMaybeConsolidation`
17+
- `WithConsolidatedOptions``ConsolidationOptions`
18+
- `BatchedRangeStoreOptions``RangeBatchingOptions`
19+
- `Stats``RangeBatchingStats`
20+
21+
**Migration:**
22+
23+
The previous exports are still available but deprecated. Update your imports:
24+
25+
```ts
26+
// Before
27+
import { withConsolidated, tryWithConsolidated, withRangeBatching } from "zarrita";
28+
29+
// After
30+
import { withConsolidation, withMaybeConsolidation, withRangeBatching } from "zarrita";
31+
32+
// Pipeline composition
33+
let store = await createStore(
34+
new FetchStore("https://..."),
35+
withConsolidation({ format: "v3" }),
36+
withRangeBatching(),
37+
);
38+
```
39+
40+
**Defining custom middleware:**
41+
42+
```ts
43+
import { wrapStore } from "zarrita";
44+
45+
const withCaching = wrapStore(
46+
(store, opts: { maxSize?: number }) => {
47+
let cache = new Map();
48+
return {
49+
async get(key, options) {
50+
let hit = cache.get(key);
51+
if (hit) return hit;
52+
let result = await store.get(key, options);
53+
if (result) cache.set(key, result);
54+
return result;
55+
},
56+
clear() { cache.clear(); },
57+
};
58+
},
59+
);
60+
```
61+
62+
`BatchedRangeStore` class has been removed in favor of `withRangeBatching` built on `wrapStore`.

packages/zarrita/__tests__/batched-fetch.test.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
import type { AbsolutePath, RangeQuery } from "@zarrita/storage";
22
import { describe, expect, it, vi } from "vitest";
3-
import { withRangeBatching } from "../src/batched-fetch.js";
3+
import { withRangeBatching } from "../src/middleware/batched-fetch.js";
44

55
/**
66
* Create a fake store with controllable getRange.

packages/zarrita/__tests__/consolidated.test.ts

Lines changed: 24 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -2,17 +2,20 @@ import * as path from "node:path";
22
import * as url from "node:url";
33
import { FileSystemStore } from "@zarrita/storage";
44
import { assert, describe, expect, it } from "vitest";
5-
import { tryWithConsolidated, withConsolidated } from "../src/consolidated.js";
65
import { NodeNotFoundError } from "../src/errors.js";
76
import { Array as ZarrArray } from "../src/hierarchy.js";
7+
import {
8+
withConsolidation,
9+
withMaybeConsolidation,
10+
} from "../src/middleware/consolidated.js";
811
import { open } from "../src/open.js";
912

1013
let __dirname = path.dirname(url.fileURLToPath(import.meta.url));
1114

12-
describe("withConsolidated", () => {
15+
describe("withConsolidation", () => {
1316
it("loads consolidated metadata", async () => {
1417
let root = path.join(__dirname, "../../../fixtures/v2/data.zarr");
15-
let store = await withConsolidated(new FileSystemStore(root));
18+
let store = await withConsolidation(new FileSystemStore(root));
1619
let map = new Map(store.contents().map((x) => [x.path, x.kind]));
1720
expect(map).toMatchInlineSnapshot(`
1821
Map {
@@ -50,7 +53,7 @@ describe("withConsolidated", () => {
5053

5154
it("loads chunk data from underlying store", async () => {
5255
let root = path.join(__dirname, "../../../fixtures/v2/data.zarr");
53-
let store = await withConsolidated(new FileSystemStore(root));
56+
let store = await withConsolidation(new FileSystemStore(root));
5457
// biome-ignore lint/style/noNonNullAssertion: Fine for a test
5558
let entry = store
5659
.contents()
@@ -88,7 +91,7 @@ describe("withConsolidated", () => {
8891

8992
it("loads and navigates from root", async () => {
9093
let path_root = path.join(__dirname, "../../../fixtures/v2/data.zarr");
91-
let store = await withConsolidated(new FileSystemStore(path_root));
94+
let store = await withConsolidation(new FileSystemStore(path_root));
9295
let grp = await open(store, { kind: "group" });
9396
expect(grp.kind).toBe("group");
9497
let arr = await open(grp.resolve("1d.chunked.i2"), { kind: "array" });
@@ -101,22 +104,22 @@ describe("withConsolidated", () => {
101104
"../../../fixtures/v2/data.zarr/3d.contiguous.i2",
102105
);
103106
let try_open = () =>
104-
withConsolidated(new FileSystemStore(root), { format: "v2" });
107+
withConsolidation(new FileSystemStore(root), { format: "v2" });
105108
await expect(try_open).rejects.toThrowError(NodeNotFoundError);
106109
await expect(try_open).rejects.toThrowErrorMatchingInlineSnapshot(
107110
"[NodeNotFoundError: Node not found: v2 consolidated metadata]",
108111
);
109112
});
110113
});
111114

112-
describe("withConsolidated (v3)", () => {
115+
describe("withConsolidation (v3)", () => {
113116
let v3root = path.join(
114117
__dirname,
115118
"../../../fixtures/v3/data.zarr/consolidated",
116119
);
117120

118121
it("loads v3 consolidated metadata", async () => {
119-
let store = await withConsolidated(new FileSystemStore(v3root), {
122+
let store = await withConsolidation(new FileSystemStore(v3root), {
120123
format: "v3",
121124
});
122125
let map = new Map(store.contents().map((x) => [x.path, x.kind]));
@@ -132,7 +135,7 @@ describe("withConsolidated (v3)", () => {
132135
});
133136

134137
it("loads chunk data from underlying store", async () => {
135-
let store = await withConsolidated(new FileSystemStore(v3root), {
138+
let store = await withConsolidation(new FileSystemStore(v3root), {
136139
format: "v3",
137140
});
138141
let grp = await open(store, { kind: "group" });
@@ -155,7 +158,7 @@ describe("withConsolidated (v3)", () => {
155158
});
156159

157160
it("loads and navigates from root", async () => {
158-
let store = await withConsolidated(new FileSystemStore(v3root), {
161+
let store = await withConsolidation(new FileSystemStore(v3root), {
159162
format: "v3",
160163
});
161164
let grp = await open(store, { kind: "group" });
@@ -168,18 +171,18 @@ describe("withConsolidated (v3)", () => {
168171
it("throws if v3 consolidated metadata is missing", async () => {
169172
let root = path.join(__dirname, "../../../fixtures/v2/data.zarr");
170173
let try_open = () =>
171-
withConsolidated(new FileSystemStore(root), { format: "v3" });
174+
withConsolidation(new FileSystemStore(root), { format: "v3" });
172175
await expect(try_open).rejects.toThrowError(NodeNotFoundError);
173176
});
174177
});
175178

176-
describe("withConsolidated (format array)", () => {
179+
describe("withConsolidation (format array)", () => {
177180
it("tries formats in order", async () => {
178181
let root = path.join(
179182
__dirname,
180183
"../../../fixtures/v3/data.zarr/consolidated",
181184
);
182-
let store = await withConsolidated(new FileSystemStore(root), {
185+
let store = await withConsolidation(new FileSystemStore(root), {
183186
format: ["v3", "v2"],
184187
});
185188
let contents = store.contents();
@@ -190,18 +193,18 @@ describe("withConsolidated (format array)", () => {
190193

191194
it("falls back to second format", async () => {
192195
let root = path.join(__dirname, "../../../fixtures/v2/data.zarr");
193-
let store = await withConsolidated(new FileSystemStore(root), {
196+
let store = await withConsolidation(new FileSystemStore(root), {
194197
format: ["v3", "v2"],
195198
});
196199
let contents = store.contents();
197200
expect(contents.length).toBeGreaterThan(0);
198201
});
199202
});
200203

201-
describe("tryWithConsolidated", () => {
204+
describe("withMaybeConsolidation", () => {
202205
it("creates Listable from consolidated store", async () => {
203206
let root = path.join(__dirname, "../../../fixtures/v2/data.zarr");
204-
let store = await tryWithConsolidated(new FileSystemStore(root));
207+
let store = await withMaybeConsolidation(new FileSystemStore(root));
205208
expect(store).toHaveProperty("contents");
206209
});
207210

@@ -210,21 +213,21 @@ describe("tryWithConsolidated", () => {
210213
__dirname,
211214
"../../../fixtures/v2/data.zarr/3d.contiguous.i2",
212215
);
213-
let store = await tryWithConsolidated(new FileSystemStore(root));
216+
let store = await withMaybeConsolidation(new FileSystemStore(root));
214217
expect(store).toBeInstanceOf(FileSystemStore);
215218
});
216219

217220
it("supports a zmetadataKey option", async () => {
218221
let root = path.join(__dirname, "../../../fixtures/v2/data.zarr");
219-
let store = await tryWithConsolidated(new FileSystemStore(root), {
222+
let store = await withMaybeConsolidation(new FileSystemStore(root), {
220223
metadataKey: ".zmetadata",
221224
});
222225
expect(store).toHaveProperty("contents");
223226
});
224227

225228
it("falls back to original store if metadataKey is incorrect", async () => {
226229
let root = path.join(__dirname, "../../../fixtures/v2/data.zarr");
227-
let store = await tryWithConsolidated(new FileSystemStore(root), {
230+
let store = await withMaybeConsolidation(new FileSystemStore(root), {
228231
metadataKey: ".nonexistent",
229232
});
230233
expect(store).toBeInstanceOf(FileSystemStore);
@@ -233,12 +236,12 @@ describe("tryWithConsolidated", () => {
233236

234237
describe("Listable.getRange", () => {
235238
it("does not expose getRange if the underlying store does not support it", async () => {
236-
let store = await tryWithConsolidated(new Map());
239+
let store = await withMaybeConsolidation(new Map());
237240
expect("getRange" in store).toBeFalsy();
238241
});
239242
it("retrieves a byte range from an underlying store", async () => {
240243
let root = path.join(__dirname, "../../../fixtures/v2/data.zarr");
241-
let store = await tryWithConsolidated(new FileSystemStore(root));
244+
let store = await withMaybeConsolidation(new FileSystemStore(root));
242245
assert(typeof store.getRange === "function");
243246
});
244247
});
Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
import type { AbsolutePath, AsyncReadable } from "@zarrita/storage";
2+
import { expectType } from "tintype";
3+
import { describe, test } from "vitest";
4+
import * as zarr from "../src/index.js";
5+
import { wrapStore } from "../src/middleware/define.js";
6+
7+
describe("createStore", () => {
8+
test("no middleware returns store as-is", () => {
9+
let store = zarr.createStore(new zarr.FetchStore(""));
10+
expectType(store).toMatchInlineSnapshot(`Promise<zarr.FetchStore>`);
11+
});
12+
});
13+
14+
describe("wrapStore", () => {
15+
test("simple: extensions appear, store methods preserved", () => {
16+
let withCustom = wrapStore(
17+
(store: AsyncReadable, _opts: { flag: boolean }) => {
18+
return {
19+
async get(key: AbsolutePath) {
20+
return store.get(key);
21+
},
22+
hello(): string {
23+
return "world";
24+
},
25+
};
26+
},
27+
);
28+
let store = withCustom(new zarr.FetchStore(""), { flag: true });
29+
expectType(store).toMatchInlineSnapshot(
30+
`zarr.FetchStore & { hello: () => string }`,
31+
);
32+
});
33+
34+
test("generic: store Options flows into opts parameter", () => {
35+
interface ThingOptions<O> {
36+
storeOptions?: O;
37+
retries?: number;
38+
}
39+
40+
let withThing = wrapStore(
41+
<O>(store: AsyncReadable<O>, opts: ThingOptions<O>) => {
42+
return {
43+
async get(key: AbsolutePath, options?: O) {
44+
return store.get(key, options ?? opts.storeOptions);
45+
},
46+
retries: opts.retries ?? 3,
47+
};
48+
},
49+
);
50+
51+
let store = withThing(new zarr.FetchStore(""), {
52+
storeOptions: { signal: AbortSignal.timeout(1000) },
53+
retries: 5,
54+
});
55+
expectType(store).toMatchInlineSnapshot(
56+
`zarr.FetchStore & { retries: number }`,
57+
);
58+
});
59+
60+
test("chaining preserves Options through wrappers", () => {
61+
let withA = wrapStore(
62+
(store: AsyncReadable, _opts: { a: number }) => {
63+
return {
64+
async get(key: AbsolutePath) {
65+
return store.get(key);
66+
},
67+
methodA(): number {
68+
return 1;
69+
},
70+
};
71+
},
72+
);
73+
let withB = wrapStore(
74+
(store: AsyncReadable, _opts: { b: string }) => {
75+
return {
76+
async get(key: AbsolutePath) {
77+
return store.get(key);
78+
},
79+
methodB(): string {
80+
return "hello";
81+
},
82+
};
83+
},
84+
);
85+
let store = withB(withA(new zarr.FetchStore(""), { a: 1 }), { b: "x" });
86+
expectType(store).toMatchInlineSnapshot(`
87+
zarr.FetchStore & { methodA: () => number } & {
88+
methodB: () => string;
89+
}
90+
`);
91+
});
92+
});

0 commit comments

Comments
 (0)