Add v3 consolidated metadata support to withConsolidated

.changeset/add-v3-consolidated.md

@@ -0,0 +1,14 @@
+---
+"zarrita": minor
+---
+
+Add v3 consolidated metadata support to `withConsolidated`. The v3 format reads `consolidated_metadata` from the root `zarr.json`, matching zarr-python's implementation. Note that v3 consolidated metadata is not yet part of the official Zarr v3 spec and should be considered experimental.
+
+A new `format` option controls which format(s) to try, accepting a single string or an array for fallback ordering. When omitted, format is auto-detected using the store's version history.
+
+```ts
+await withConsolidated(store);                            // auto-detect
+await withConsolidated(store, { format: "v2" });          // v2 only
+await withConsolidated(store, { format: "v3" });          // v3 only
+await withConsolidated(store, { format: ["v3", "v2"] });  // try v3, fall back to v2
+```

docs/cookbook.md

@@ -88,17 +88,16 @@ const group = await zarr.create(store, {
 group; // zarr.Group
 ```
 
-## Open a Consolidated Store
+## Open a Consolidated Store <Badge type="tip" text="v2 & v3" />
 
-Zarr v2 allows
-[consolidating metadata](https://zarr.readthedocs.io/en/stable/tutorial.html#consolidating-metadata)
-for an entire hierarchy at the store root (under `/.zmetadata`). Metadata
-consolidation is particularly useful when interacting with remote stores, where
-each metadata fetch incurs a network request and hence, latency.
+Consolidated metadata stores the entire hierarchy's metadata in a single
+location, avoiding per-node network requests. This is particularly useful
+with remote stores where each metadata fetch incurs latency.
 
 The `withConsolidated` helper wraps an existing [store](/packages/storage),
 proxying metadata requests with the consolidated metadata, thereby minimizing
-network requests.
+network requests. It supports both Zarr v2 (`.zmetadata`) and v3
+(`consolidated_metadata` in root `zarr.json`).
 
 ```js{3}
 import * as zarr from "zarrita";
@@ -113,15 +112,29 @@ let foo = await zarr.open(root.resolve("foo"), { kind: "array" });
 ```
 
 The store returned from `withConsolidated` is **readonly** and adds
-`.contents()` list the known contents of the hierarchy:
+`.contents()` to list the known contents of the hierarchy:
 
 ```js
 store.contents(); // [{ path: "/", kind: "group" }, { path: "/foo", kind: "array" }, ...]
 ```
 
+By default, the format is auto-detected. You can explicitly specify the
+format with the `format` option, or provide an array to try formats in order:
+
+```js
+// v2 only
+let store = await zarr.withConsolidated(rawStore, { format: "v2" });
+
+// v3 only
+let store = await zarr.withConsolidated(rawStore, { format: "v3" });
+
+// try v3 first, fall back to v2
+let store = await zarr.withConsolidated(rawStore, { format: ["v3", "v2"] });
+```
+
 ::: tip
 
-The `withConsolidated` helper errors out if v2 consolidated metadata is absent.
+The `withConsolidated` helper errors out if consolidated metadata is absent.
 Use `tryWithConsolidated` for uncertain cases; it leverages consolidated
 metadata if available.
 
@@ -133,6 +146,15 @@ let store = await zarr.tryWithConsolidated(
 
 :::
 
+::: warning
+
+Zarr v3 consolidated metadata (`format: "v3"`) targets the experimental
+consolidated metadata implemented in zarr-python, which is not yet part of the
+official Zarr v3 specification. See [zarr-specs#309](https://github.com/zarr-developers/zarr-specs/pull/309)
+for the ongoing spec discussion.
+
+:::
+
 
 ## Batch and Cache Range Requests
 

fixtures/v3/data.zarr/consolidated/1d.chunked.i2/zarr.json

@@ -0,0 +1,33 @@
+{
+  "shape": [
+    4
+  ],
+  "data_type": "int16",
+  "chunk_grid": {
+    "name": "regular",
+    "configuration": {
+      "chunk_shape": [
+        2
+      ]
+    }
+  },
+  "chunk_key_encoding": {
+    "name": "default",
+    "configuration": {
+      "separator": "/"
+    }
+  },
+  "fill_value": 0,
+  "codecs": [
+    {
+      "name": "bytes",
+      "configuration": {
+        "endian": "little"
+      }
+    }
+  ],
+  "attributes": {},
+  "zarr_format": 3,
+  "node_type": "array",
+  "storage_transformers": []
+}

fixtures/v3/data.zarr/consolidated/2d.contiguous.i2/zarr.json

@@ -0,0 +1,35 @@
+{
+  "shape": [
+    2,
+    2
+  ],
+  "data_type": "int16",
+  "chunk_grid": {
+    "name": "regular",
+    "configuration": {
+      "chunk_shape": [
+        2,
+        2
+      ]
+    }
+  },
+  "chunk_key_encoding": {
+    "name": "default",
+    "configuration": {
+      "separator": "/"
+    }
+  },
+  "fill_value": 0,
+  "codecs": [
+    {
+      "name": "bytes",
+      "configuration": {
+        "endian": "little"
+      }
+    }
+  ],
+  "attributes": {},
+  "zarr_format": 3,
+  "node_type": "array",
+  "storage_transformers": []
+}

fixtures/v3/data.zarr/consolidated/nested/1d.i2/zarr.json

@@ -0,0 +1,33 @@
+{
+  "shape": [
+    4
+  ],
+  "data_type": "int16",
+  "chunk_grid": {
+    "name": "regular",
+    "configuration": {
+      "chunk_shape": [
+        4
+      ]
+    }
+  },
+  "chunk_key_encoding": {
+    "name": "default",
+    "configuration": {
+      "separator": "/"
+    }
+  },
+  "fill_value": 0,
+  "codecs": [
+    {
+      "name": "bytes",
+      "configuration": {
+        "endian": "little"
+      }
+    }
+  ],
+  "attributes": {},
+  "zarr_format": 3,
+  "node_type": "array",
+  "storage_transformers": []
+}

fixtures/v3/data.zarr/consolidated/nested/zarr.json

@@ -0,0 +1,8 @@
+{
+  "attributes": {
+    "description": "A nested group"
+  },
+  "zarr_format": 3,
+  "consolidated_metadata": null,
+  "node_type": "group"
+}

fixtures/v3/data.zarr/consolidated/zarr.json

@@ -0,0 +1,126 @@
+{
+  "attributes": {
+    "answer": 42
+  },
+  "zarr_format": 3,
+  "consolidated_metadata": {
+    "kind": "inline",
+    "must_understand": false,
+    "metadata": {
+      "1d.chunked.i2": {
+        "shape": [
+          4
+        ],
+        "data_type": "int16",
+        "chunk_grid": {
+          "name": "regular",
+          "configuration": {
+            "chunk_shape": [
+              2
+            ]
+          }
+        },
+        "chunk_key_encoding": {
+          "name": "default",
+          "configuration": {
+            "separator": "/"
+          }
+        },
+        "fill_value": 0,
+        "codecs": [
+          {
+            "name": "bytes",
+            "configuration": {
+              "endian": "little"
+            }
+          }
+        ],
+        "attributes": {},
+        "zarr_format": 3,
+        "node_type": "array",
+        "storage_transformers": []
+      },
+      "2d.contiguous.i2": {
+        "shape": [
+          2,
+          2
+        ],
+        "data_type": "int16",
+        "chunk_grid": {
+          "name": "regular",
+          "configuration": {
+            "chunk_shape": [
+              2,
+              2
+            ]
+          }
+        },
+        "chunk_key_encoding": {
+          "name": "default",
+          "configuration": {
+            "separator": "/"
+          }
+        },
+        "fill_value": 0,
+        "codecs": [
+          {
+            "name": "bytes",
+            "configuration": {
+              "endian": "little"
+            }
+          }
+        ],
+        "attributes": {},
+        "zarr_format": 3,
+        "node_type": "array",
+        "storage_transformers": []
+      },
+      "nested": {
+        "attributes": {
+          "description": "A nested group"
+        },
+        "zarr_format": 3,
+        "consolidated_metadata": {
+          "kind": "inline",
+          "must_understand": false,
+          "metadata": {}
+        },
+        "node_type": "group"
+      },
+      "nested/1d.i2": {
+        "shape": [
+          4
+        ],
+        "data_type": "int16",
+        "chunk_grid": {
+          "name": "regular",
+          "configuration": {
+            "chunk_shape": [
+              4
+            ]
+          }
+        },
+        "chunk_key_encoding": {
+          "name": "default",
+          "configuration": {
+            "separator": "/"
+          }
+        },
+        "fill_value": 0,
+        "codecs": [
+          {
+            "name": "bytes",
+            "configuration": {
+              "endian": "little"
+            }
+          }
+        ],
+        "attributes": {},
+        "zarr_format": 3,
+        "node_type": "array",
+        "storage_transformers": []
+      }
+    }
+  },
+  "node_type": "group"
+}