Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions .changeset/add-v3-consolidated.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,13 @@
"zarrita": minor
---

Add v3 consolidated metadata support to `withConsolidated`. The v3 format reads `consolidated_metadata` from the root `zarr.json`, matching zarr-python's implementation. Note that v3 consolidated metadata is not yet part of the official Zarr v3 spec and should be considered experimental.
Add v3 consolidated metadata support to `withConsolidatedMetadata`. The v3 format reads `consolidated_metadata` from the root `zarr.json`, matching zarr-python's implementation. Note that v3 consolidated metadata is not yet part of the official Zarr v3 spec and should be considered experimental.

A new `format` option controls which format(s) to try, accepting a single string or an array for fallback ordering. When omitted, format is auto-detected using the store's version history.

```ts
await withConsolidated(store); // auto-detect
await withConsolidated(store, { format: "v2" }); // v2 only
await withConsolidated(store, { format: "v3" }); // v3 only
await withConsolidated(store, { format: ["v3", "v2"] }); // try v3, fall back to v2
await withConsolidatedMetadata(store); // auto-detect
await withConsolidatedMetadata(store, { format: "v2" }); // v2 only
await withConsolidatedMetadata(store, { format: "v3" }); // v3 only
await withConsolidatedMetadata(store, { format: ["v3", "v2"] }); // try v3, fall back to v2
```
10 changes: 5 additions & 5 deletions .changeset/store-extension-system.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,14 @@ Introduce composable store and array extensions via `defineStoreExtension` and `

- `zarr.defineStoreExtension(factory)` — define a store extension with automatic Proxy delegation. The factory receives an `AsyncReadable` store and options, returning overrides and extensions. Supports sync and async factories.
- `zarr.defineArrayExtension(factory)` — define an array extension that intercepts `getChunk` on a `zarr.Array`. Same Proxy-delegation model.
- `zarr.extendStore(store, ...extensions)` — compose store extensions in a pipeline. Each extension is `(store) => newStore`. Returns `Promise` to support async extensions like `withConsolidation`.
- `zarr.extendStore(store, ...extensions)` — compose store extensions in a pipeline. Each extension is `(store) => newStore`. Returns `Promise` to support async extensions like `withConsolidatedMetadata`.
- `zarr.extendArray(array, ...extensions)` — compose array extensions in a pipeline.

**Renamed:**

- `withConsolidated` → `withConsolidation`
- `tryWithConsolidated` → `withMaybeConsolidation`
- `WithConsolidatedOptions` → `ConsolidationOptions`
- `withConsolidated` → `withConsolidatedMetadata`
- `tryWithConsolidated` → `withMaybeConsolidatedMetadata`
- `WithConsolidatedOptions` → `ConsolidatedMetadataOptions`
- `BatchedRangeStoreOptions` → `RangeBatchingOptions`
- `Stats` → `RangeBatchingStats`

Expand All @@ -29,7 +29,7 @@ import * as zarr from "zarrita";
// Pipeline composition (use arrow functions for full type inference)
let store = await zarr.extendStore(
new zarr.FetchStore("https://..."),
zarr.withConsolidation,
zarr.withConsolidatedMetadata,
(s) => zarr.withRangeBatching(s, { mergeOptions: (batch) => batch[0] }),
);
```
Expand Down
8 changes: 4 additions & 4 deletions docs/store-extensions.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,24 +24,24 @@ import * as zarr from "zarrita";

let store = await zarr.extendStore(
new zarr.FetchStore("https://example.com/data.zarr"),
zarr.withConsolidation,
zarr.withConsolidatedMetadata,
(s) => zarr.withRangeBatching(s, { cacheSize: 512 }),
);

store.contents(); // from zarr.withConsolidation
store.contents(); // from zarr.withConsolidatedMetadata
store.stats; // from zarr.withRangeBatching
```

Each extension wraps the previous result. `zarr.extendStore` handles async
extensions (like `zarr.withConsolidation`, which fetches metadata during
extensions (like `zarr.withConsolidatedMetadata`, which fetches metadata during
initialization) automatically, and always returns a `Promise`. An extension
with no required options can be passed uncalled; otherwise wrap it in an arrow
so the options are applied to the argument.

You can also call extensions directly:

```ts
let consolidated = await zarr.withConsolidation(
let consolidated = await zarr.withConsolidatedMetadata(
new zarr.FetchStore("https://example.com/data.zarr"),
{ format: "v3" },
);
Expand Down
44 changes: 22 additions & 22 deletions packages/zarrita/__tests__/consolidated.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,18 +4,18 @@ import { FileSystemStore } from "@zarrita/storage";
import { assert, describe, expect, it } from "vitest";
import { NotFoundError } from "../src/errors.js";
import {
withConsolidation,
withMaybeConsolidation,
withConsolidatedMetadata,
withMaybeConsolidatedMetadata,
} from "../src/extension/consolidation.js";
import { Array as ZarrArray } from "../src/hierarchy.js";
import { open } from "../src/open.js";

let __dirname = path.dirname(url.fileURLToPath(import.meta.url));

describe("withConsolidation", () => {
describe("withConsolidatedMetadata", () => {
it("loads consolidated metadata", async () => {
let root = path.join(__dirname, "../../../fixtures/v2/data.zarr");
let store = await withConsolidation(new FileSystemStore(root));
let store = await withConsolidatedMetadata(new FileSystemStore(root));
let map = new Map(store.contents().map((x) => [x.path, x.kind]));
expect(map).toMatchInlineSnapshot(`
Map {
Expand Down Expand Up @@ -57,7 +57,7 @@ describe("withConsolidation", () => {

it("loads chunk data from underlying store", async () => {
let root = path.join(__dirname, "../../../fixtures/v2/data.zarr");
let store = await withConsolidation(new FileSystemStore(root));
let store = await withConsolidatedMetadata(new FileSystemStore(root));
// biome-ignore lint/style/noNonNullAssertion: Fine for a test
let entry = store
.contents()
Expand Down Expand Up @@ -95,7 +95,7 @@ describe("withConsolidation", () => {

it("loads and navigates from root", async () => {
let path_root = path.join(__dirname, "../../../fixtures/v2/data.zarr");
let store = await withConsolidation(new FileSystemStore(path_root));
let store = await withConsolidatedMetadata(new FileSystemStore(path_root));
let grp = await open(store, { kind: "group" });
expect(grp.kind).toBe("group");
let arr = await open(grp.resolve("1d.chunked.i2"), { kind: "array" });
Expand All @@ -108,22 +108,22 @@ describe("withConsolidation", () => {
"../../../fixtures/v2/data.zarr/3d.contiguous.i2",
);
let try_open = () =>
withConsolidation(new FileSystemStore(root), { format: "v2" });
withConsolidatedMetadata(new FileSystemStore(root), { format: "v2" });
await expect(try_open).rejects.toThrowError(NotFoundError);
await expect(try_open).rejects.toThrowErrorMatchingInlineSnapshot(
"[NotFoundError: Not found: v2 consolidated metadata]",
);
});
});

describe("withConsolidation (v3)", () => {
describe("withConsolidatedMetadata (v3)", () => {
let v3root = path.join(
__dirname,
"../../../fixtures/v3/data.zarr/consolidated",
);

it("loads v3 consolidated metadata", async () => {
let store = await withConsolidation(new FileSystemStore(v3root), {
let store = await withConsolidatedMetadata(new FileSystemStore(v3root), {
format: "v3",
});
let map = new Map(store.contents().map((x) => [x.path, x.kind]));
Expand All @@ -139,7 +139,7 @@ describe("withConsolidation (v3)", () => {
});

it("loads chunk data from underlying store", async () => {
let store = await withConsolidation(new FileSystemStore(v3root), {
let store = await withConsolidatedMetadata(new FileSystemStore(v3root), {
format: "v3",
});
let grp = await open(store, { kind: "group" });
Expand All @@ -162,7 +162,7 @@ describe("withConsolidation (v3)", () => {
});

it("loads and navigates from root", async () => {
let store = await withConsolidation(new FileSystemStore(v3root), {
let store = await withConsolidatedMetadata(new FileSystemStore(v3root), {
format: "v3",
});
let grp = await open(store, { kind: "group" });
Expand All @@ -175,18 +175,18 @@ describe("withConsolidation (v3)", () => {
it("throws if v3 consolidated metadata is missing", async () => {
let root = path.join(__dirname, "../../../fixtures/v2/data.zarr");
let try_open = () =>
withConsolidation(new FileSystemStore(root), { format: "v3" });
withConsolidatedMetadata(new FileSystemStore(root), { format: "v3" });
await expect(try_open).rejects.toThrowError(NotFoundError);
});
});

describe("withConsolidation (format array)", () => {
describe("withConsolidatedMetadata (format array)", () => {
it("tries formats in order", async () => {
let root = path.join(
__dirname,
"../../../fixtures/v3/data.zarr/consolidated",
);
let store = await withConsolidation(new FileSystemStore(root), {
let store = await withConsolidatedMetadata(new FileSystemStore(root), {
format: ["v3", "v2"],
});
let contents = store.contents();
Expand All @@ -197,18 +197,18 @@ describe("withConsolidation (format array)", () => {

it("falls back to second format", async () => {
let root = path.join(__dirname, "../../../fixtures/v2/data.zarr");
let store = await withConsolidation(new FileSystemStore(root), {
let store = await withConsolidatedMetadata(new FileSystemStore(root), {
format: ["v3", "v2"],
});
let contents = store.contents();
expect(contents.length).toBeGreaterThan(0);
});
});

describe("withMaybeConsolidation", () => {
describe("withMaybeConsolidatedMetadata", () => {
it("creates Listable from consolidated store", async () => {
let root = path.join(__dirname, "../../../fixtures/v2/data.zarr");
let store = await withMaybeConsolidation(new FileSystemStore(root));
let store = await withMaybeConsolidatedMetadata(new FileSystemStore(root));
expect(store).toHaveProperty("contents");
});

Expand All @@ -217,21 +217,21 @@ describe("withMaybeConsolidation", () => {
__dirname,
"../../../fixtures/v2/data.zarr/3d.contiguous.i2",
);
let store = await withMaybeConsolidation(new FileSystemStore(root));
let store = await withMaybeConsolidatedMetadata(new FileSystemStore(root));
expect(store).toBeInstanceOf(FileSystemStore);
});

it("supports a zmetadataKey option", async () => {
let root = path.join(__dirname, "../../../fixtures/v2/data.zarr");
let store = await withMaybeConsolidation(new FileSystemStore(root), {
let store = await withMaybeConsolidatedMetadata(new FileSystemStore(root), {
metadataKey: ".zmetadata",
});
expect(store).toHaveProperty("contents");
});

it("falls back to original store if metadataKey is incorrect", async () => {
let root = path.join(__dirname, "../../../fixtures/v2/data.zarr");
let store = await withMaybeConsolidation(new FileSystemStore(root), {
let store = await withMaybeConsolidatedMetadata(new FileSystemStore(root), {
metadataKey: ".nonexistent",
});
expect(store).toBeInstanceOf(FileSystemStore);
Expand All @@ -240,12 +240,12 @@ describe("withMaybeConsolidation", () => {

describe("Listable.getRange", () => {
it("does not expose getRange if the underlying store does not support it", async () => {
let store = await withMaybeConsolidation(new Map());
let store = await withMaybeConsolidatedMetadata(new Map());
expect("getRange" in store).toBeFalsy();
});
it("retrieves a byte range from an underlying store", async () => {
let root = path.join(__dirname, "../../../fixtures/v2/data.zarr");
let store = await withMaybeConsolidation(new FileSystemStore(root));
let store = await withMaybeConsolidatedMetadata(new FileSystemStore(root));
assert(typeof store.getRange === "function");
});
});
2 changes: 1 addition & 1 deletion packages/zarrita/__tests__/extension-types.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ describe("extendStore", () => {
function check() {
return zarr.extendStore(
new zarr.FetchStore(""),
zarr.withConsolidation,
zarr.withConsolidatedMetadata,
(s) => zarr.withRangeBatching(s),
);
}
Expand Down
6 changes: 2 additions & 4 deletions packages/zarrita/__tests__/public-api.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -35,10 +35,8 @@ test("public API surface", () => {
"select",
"set",
"slice",
"tryWithConsolidated",
"withConsolidated",
"withConsolidation",
"withMaybeConsolidation",
"withConsolidatedMetadata",
"withMaybeConsolidatedMetadata",
"withRangeBatching",
]
`);
Expand Down
34 changes: 18 additions & 16 deletions packages/zarrita/src/extension/consolidation.ts
Original file line number Diff line number Diff line change
Expand Up @@ -67,8 +67,8 @@ function isConsolidatedV3(meta: unknown): meta is GroupMetadata & {
/** The format of consolidated metadata to use. */
export type ConsolidatedFormat = "v2" | "v3";

/** Options for {@linkcode withConsolidation} and {@linkcode withMaybeConsolidation}. */
export interface ConsolidationOptions {
/** Options for {@linkcode withConsolidatedMetadata} and {@linkcode withMaybeConsolidatedMetadata}. */
export interface ConsolidatedMetadataOptions {
/**
* The format(s) of consolidated metadata to try.
*
Expand Down Expand Up @@ -197,22 +197,22 @@ export type Listable<Store extends Readable> = Store & {
* @example
* ```ts
* // Direct
* let store = await zarr.withConsolidation(new zarr.FetchStore("https://..."));
* let store = await zarr.withConsolidatedMetadata(new zarr.FetchStore("https://..."));
*
* // With options
* let store = await zarr.withConsolidation(rawStore, { format: "v3" });
* let store = await zarr.withConsolidatedMetadata(rawStore, { format: "v3" });
*
* // In a pipeline
* let store = await zarr.extendStore(
* new zarr.FetchStore("https://..."),
* (s) => zarr.withConsolidation(s, { format: "v3" }),
* (s) => zarr.withConsolidatedMetadata(s, { format: "v3" }),
* );
*
* store.contents(); // [{ path: "/", kind: "group" }, ...]
* ```
*/
export const withConsolidation = defineStoreExtension(
async (store, opts: ConsolidationOptions = {}) => {
export const withConsolidatedMetadata = defineStoreExtension(
async (store, opts: ConsolidatedMetadataOptions = {}) => {
let formats = resolveFormats(store, opts.format);
let lastError: unknown;
for (let format of formats) {
Expand Down Expand Up @@ -265,17 +265,19 @@ export const withConsolidation = defineStoreExtension(
);

/**
* Like {@linkcode withConsolidation}, but falls back to the original store if
* Like {@linkcode withConsolidatedMetadata}, but falls back to the original store if
* no consolidated metadata is found (instead of throwing).
*/
export async function withMaybeConsolidation<Store extends AsyncReadable>(
export async function withMaybeConsolidatedMetadata<
Store extends AsyncReadable,
>(
store: Store,
opts: ConsolidationOptions = {},
opts: ConsolidatedMetadataOptions = {},
): Promise<Listable<Store> | Store> {
return (withConsolidation(store, opts) as Promise<Listable<Store>>).catch(
(error: unknown) => {
rethrowUnless(error, NotFoundError, InvalidMetadataError);
return store;
},
);
return (
withConsolidatedMetadata(store, opts) as Promise<Listable<Store>>
).catch((error: unknown) => {
rethrowUnless(error, NotFoundError, InvalidMetadataError);
return store;
});
}
13 changes: 3 additions & 10 deletions packages/zarrita/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,21 +14,14 @@ export {
UnknownCodecError,
UnsupportedError,
} from "./errors.js";
/** @deprecated Use {@linkcode ConsolidationOptions} instead. */
export type {
ConsolidatedFormat,
ConsolidationOptions,
ConsolidationOptions as WithConsolidatedOptions,
ConsolidatedMetadataOptions,
Listable,
} from "./extension/consolidation.js";
// deprecated re-exports
/** @deprecated Use {@linkcode withConsolidation} instead. */
/** @deprecated Use {@linkcode withMaybeConsolidation} instead. */
export {
withConsolidation,
withConsolidation as withConsolidated,
withMaybeConsolidation,
withMaybeConsolidation as tryWithConsolidated,
withConsolidatedMetadata,
withMaybeConsolidatedMetadata,
} from "./extension/consolidation.js";
export { defineStoreExtension } from "./extension/define.js";
export {
Expand Down