Skip to content

Commit 858cf99

Browse files
committed
Add v3 consolidated metadata support to withConsolidated
`withConsolidated` previously only supported Zarr v2 `.zmetadata`. This adds support for v3 consolidated metadata (inline `consolidated_metadata` in the root `zarr.json`), which is implemented in zarr-python but not yet accepted into the Zarr v3 spec. A new `format` option controls which format(s) to try. It accepts a single format string or an array to try in order. When omitted, auto- detection uses the existing version counter heuristic from `open`. ```ts await withConsolidated(store); // auto-detect await withConsolidated(store, { format: "v2" }); // v2 only await withConsolidated(store, { format: "v3" }); // v3 only await withConsolidated(store, { format: ["v3", "v2"] }); // try v3, fall back to v2 ```
1 parent 164370d commit 858cf99

9 files changed

Lines changed: 6394 additions & 74 deletions

File tree

.changeset/add-v3-consolidated.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
---
2+
"zarrita": minor
3+
---
4+
5+
Add v3 consolidated metadata support to `withConsolidated`. The v3 format reads `consolidated_metadata` from the root `zarr.json`, matching zarr-python's implementation. Note that v3 consolidated metadata is not yet part of the official Zarr v3 spec and should be considered experimental.
6+
7+
A new `format` option controls which format(s) to try, accepting a single string or an array for fallback ordering. When omitted, format is auto-detected using the store's version history.
8+
9+
```ts
10+
await withConsolidated(store); // auto-detect
11+
await withConsolidated(store, { format: "v2" }); // v2 only
12+
await withConsolidated(store, { format: "v3" }); // v3 only
13+
await withConsolidated(store, { format: ["v3", "v2"] }); // try v3, fall back to v2
14+
```

docs/cookbook.md

Lines changed: 31 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -88,17 +88,16 @@ const group = await zarr.create(store, {
8888
group; // zarr.Group
8989
```
9090

91-
## Open a Consolidated Store
91+
## Open a Consolidated Store <Badge type="tip" text="v2 & v3" />
9292

93-
Zarr v2 allows
94-
[consolidating metadata](https://zarr.readthedocs.io/en/stable/tutorial.html#consolidating-metadata)
95-
for an entire hierarchy at the store root (under `/.zmetadata`). Metadata
96-
consolidation is particularly useful when interacting with remote stores, where
97-
each metadata fetch incurs a network request and hence, latency.
93+
Consolidated metadata stores the entire hierarchy's metadata in a single
94+
location, avoiding per-node network requests. This is particularly useful
95+
with remote stores where each metadata fetch incurs latency.
9896

9997
The `withConsolidated` helper wraps an existing [store](/packages/storage),
10098
proxying metadata requests with the consolidated metadata, thereby minimizing
101-
network requests.
99+
network requests. It supports both Zarr v2 (`.zmetadata`) and v3
100+
(`consolidated_metadata` in root `zarr.json`).
102101

103102
```js{3}
104103
import * as zarr from "zarrita";
@@ -113,15 +112,29 @@ let foo = await zarr.open(root.resolve("foo"), { kind: "array" });
113112
```
114113

115114
The store returned from `withConsolidated` is **readonly** and adds
116-
`.contents()` list the known contents of the hierarchy:
115+
`.contents()` to list the known contents of the hierarchy:
117116

118117
```js
119118
store.contents(); // [{ path: "/", kind: "group" }, { path: "/foo", kind: "array" }, ...]
120119
```
121120

121+
By default, the format is auto-detected. You can explicitly specify the
122+
format with the `format` option, or provide an array to try formats in order:
123+
124+
```js
125+
// v2 only
126+
let store = await zarr.withConsolidated(rawStore, { format: "v2" });
127+
128+
// v3 only
129+
let store = await zarr.withConsolidated(rawStore, { format: "v3" });
130+
131+
// try v3 first, fall back to v2
132+
let store = await zarr.withConsolidated(rawStore, { format: ["v3", "v2"] });
133+
```
134+
122135
::: tip
123136

124-
The `withConsolidated` helper errors out if v2 consolidated metadata is absent.
137+
The `withConsolidated` helper errors out if consolidated metadata is absent.
125138
Use `tryWithConsolidated` for uncertain cases; it leverages consolidated
126139
metadata if available.
127140

@@ -133,6 +146,15 @@ let store = await zarr.tryWithConsolidated(
133146

134147
:::
135148

149+
::: warning
150+
151+
Zarr v3 consolidated metadata (`format: "v3"`) targets the experimental
152+
consolidated metadata implemented in zarr-python, which is not yet part of the
153+
official Zarr v3 specification. See [zarr-specs#309](https://github.com/zarr-developers/zarr-specs/pull/309)
154+
for the ongoing spec discussion.
155+
156+
:::
157+
136158

137159
## Batch and Cache Range Requests
138160

0 commit comments

Comments
 (0)