feat(zarr-metadata): composition rules layer, shape-exact entity validators, create_* factories

packages/zarr-metadata/README.md

@@ -24,21 +24,27 @@ Two layers and an optional integration:
 ## What this is for
 
 The public `TypedDict` definitions describe the static JSON shape of Zarr
-metadata. For strict, loc-aware validation of JSON loaded from disk, use the
-model parser:
+metadata. To judge JSON loaded from disk, structure and composition together,
+use the rules layer; to get a normalized document model, use the model parser:
 
 ```python
 import json
 from zarr_metadata.model import ZarrV3ArrayMetadata
+from zarr_metadata.rules import parse_array_metadata_v3
 
 with open("zarr.json", "rb") as f:
     raw = json.load(f)
 
-metadata = ZarrV3ArrayMetadata.from_json(raw)
+document = parse_array_metadata_v3(raw)  # raises with every problem found
+metadata = ZarrV3ArrayMetadata.from_json(document)
 ```
 
-The optional Pydantic integration delegates raw input to the same strict
-parser and returns the same normalized model class:
+To construct a document, the `create_*` factories in `zarr_metadata.builder`
+apply the same judgment to keyword arguments typed by the document's
+`TypedDict`.
+
+The optional Pydantic integration runs raw input through the rules layer
+and returns the same normalized model class:
 
 ```python
 from pydantic import TypeAdapter
@@ -56,19 +62,28 @@ members that the strict model parser rejects.
 
 The model validators enforce the declared document structure and a small set
 of context-free consistency rules, including fixed format literals, finite
-JSON numbers, non-negative dimensions, non-empty v3 codec pipelines, and one
-`dimension_names` entry per array dimension. They do not interpret extension
-names or configurations, resolve codec pipelines, or decide whether a data
-type, chunk grid, codec, or storage transformer is supported. Those decisions
-belong to consumer implementations.
+JSON numbers, non-negative dimensions, and non-empty v3 codec pipelines.
+They do not interpret extension names or configurations.
+
+The composition rules (`zarr_metadata.rules`) judge the document as a whole:
+fill values against data types, codec pipeline ordering, chunk-grid
+geometry against `shape`, one `dimension_names` entry per array dimension,
+and the canonical configuration shapes of the codecs, chunk grids, chunk
+key encodings, and data types this package defines. Unknown extension names
+are left unjudged. The rules model canonical documents and are deliberately
+stricter than any given implementation: an implementation may coerce
+ambiguous input as it sees fit and then validate the canonical result.
+Nothing here decides whether a data type, chunk grid, codec, or storage
+transformer is *supported*; that belongs to consumer implementations.
 
 The Pydantic integration's generated JSON Schemas express independently
 checkable document structure and field constraints, but they are not a
 replacement for runtime model validation. Standard JSON Schema treats a
 mathematically integral number such as `1.0` as an integer, while the runtime
 boundary requires Python `int` values, and it cannot express arbitrary
 same-length relations such as `dimension_names` versus `shape` or v2 `chunks`
-versus `shape`. Consumers should run the model parser after schema validation.
+versus `shape`. Consumers should run the runtime validators after schema
+validation.
 
 ## Scope
 

packages/zarr-metadata/changes/318.feature.3.md

@@ -0,0 +1,14 @@
+Added `create_*` factories in `zarr_metadata.builder`, one per public
+document TypedDict (`create_zarr_v3_array_metadata_json`,
+`create_zarr_v3_group_metadata_json`, `create_zarr_v3_consolidated_metadata_json`,
+`create_zarr_v2_array_metadata_json`, `create_zarr_v2_group_metadata_json`,
+`create_zarr_v2_zarray_json`, `create_zarr_v2_zgroup_json`,
+`create_zarr_v2_consolidated_metadata_json`), each taking
+`**kwargs: Unpack[<TypedDict>]`. Each factory copies and normalizes its
+input, runs structural and composition validation, and raises one
+`MetadataValidationError` containing all problems. The strict on-disk
+`.zarray`/`.zgroup` factories reject `attributes` at runtime, and the v2
+consolidated factory validates each entry against the document shape its
+path suffix selects. The open v3 array/group factories take an
+`extensions=` mapping for extension fields (for type checkers without PEP
+728 support) and reject names that shadow standard fields.

packages/zarr-metadata/changes/318.feature.5.md

@@ -0,0 +1,17 @@
+Added `check_*` entry points in `zarr_metadata.rules` returning a
+discriminated `Valid[T] | Invalid`, for callers who want a document and
+its problems in one value.
+
+The literal `valid` field narrows to either the normalized document or a
+nonempty problem tuple:
+
+```python
+result = check_array_metadata_v3(loaded)
+if result.valid:
+    store(result.document)  # typed ZarrV3ArrayMetadataJSON
+else:
+    report(result.problems)  # non-empty tuple of problems
+```
+
+Use `validate_*` to collect problems and `parse_*` to raise on invalid
+input.

packages/zarr-metadata/changes/318.feature.6.md

@@ -0,0 +1,13 @@
+Unknown members inside a *known* entity's `configuration` (e.g. an extra
+key in a `blosc` configuration) now report as their own `unknown_key`
+problem kind, and no longer suppress the other rules about that entity.
+
+Whether configurations are closed remains unspecified
+([zarr-specs#270](https://github.com/zarr-developers/zarr-specs/issues/270)),
+so this package retains its strict reading with two safeguards:
+
+- callers can filter the dedicated `unknown_key` kind;
+- unknown keys do not suppress other rules for the same entity.
+
+Model round-trips preserve unmodeled members. Shape-exact `TypeIs` guards
+still reject them because the corresponding TypedDicts are closed.

packages/zarr-metadata/changes/318.feature.md

@@ -0,0 +1,51 @@
+Added `zarr_metadata.rules`: composition rules for full metadata
+documents. The package now models metadata in three layers with one
+contract each — `model` checks structure element by element, `rules`
+judges composition across the document, and `builder` constructs while
+applying both. Rules are registered where they are defined; rules about a
+particular codec, chunk grid, or data type live with that entity under
+`rules._entities` and are dispatched by name, so adding an entity adds a
+module there and changes nothing else.
+
+- **Rule sets**: `ZARR_V3_ARRAY_RULES` covers fill value vs. data type,
+  codec pipeline kind ordering, known-name shapes,
+  dimension-name counts, chunk-grid values (positive extents) and
+  geometry (regular rank; rectilinear rank and per-dimension chunk-size
+  sums, RLE pairs included), transpose orders (self-permutation at any
+  depth, rank agreement with `shape`), and sharding (inner `codecs` and
+  `index_codecs` judged as pipelines recursively at every nesting depth;
+  inner chunk shapes positive, rank-matched, and evenly dividing the
+  enclosing chunk, recursively). New `ZARR_V2_ARRAY_RULES`
+  (chunks/shape rank agreement) and `ZARR_V3_GROUP_RULES` (inline
+  consolidated metadata recurses, judging each embedded child document
+  by its own rules at its path).
+- **Read-side trios**: `validate_*` / `is_*` / `parse_*` for array and
+  group documents in both format versions mirror the model layer's
+  grammar with a stronger judgment — structure *and* composition, every
+  problem reported together, JSON arrays normalized to tuples before
+  judgment. The `is_*` functions deliberately return `bool` rather than
+  `TypeIs`: a composition-invalid document is still an instance of the
+  TypedDict, so only the structural layer can narrow honestly.
+- **Boundary change**: two composition checks that lived in the
+  structural validator moved here — v3 `dimension_names` vs `shape` and
+  v2 `chunks` vs `shape` rank agreement. `zarr_metadata.model`'s
+  validators, parsers, and dataclasses now accept those documents (they
+  are lossless, structurally well-formed representations of what a store
+  may contain); use the `rules` trios to judge them. This also removes
+  the double report the overlap used to produce.
+- **Strictness stance**, now documented on the package: `zarr_metadata`
+  models canonical documents and is deliberately stricter than any given
+  implementation; implementations coerce ambiguous input as they see fit
+  and then validate the canonical result.
+
+- **Codec chains are judged against the array each codec receives**:
+  `transpose` permutes the shape and `cast_value` changes the data type
+  seen by everything after it, so a shard behind a transpose must divide
+  the transposed chunk, and a `bytes` codec behind a cast needs an
+  endianness for the *target* type. `zarr_metadata.v3.codec.kind` sorts
+  known codec names into the spec's three pipeline kinds.
+- **Pydantic field types** for array and group documents now run the
+  composition rules as well as structural validation.
+
+Known follow-up: v2 fill-value/dtype consistency (NumPy dtype grammar)
+has no rule yet.

packages/zarr-metadata/docs/api/builder.md

@@ -0,0 +1,5 @@
+---
+title: builder
+---
+
+::: zarr_metadata.builder

packages/zarr-metadata/docs/api/index.md

@@ -8,6 +8,12 @@ The package is organized to mirror the structure of the Zarr specifications:
 
 - [`zarr_metadata.model`](model.md) — frozen-dataclass document models,
   structural validators, loc-aware parsers, and the `UNSET` sentinel
+- [`zarr_metadata.rules`](rules.md) — composition rules: cross-field
+  judgments over full documents (fill value vs. data type, codec pipeline
+  ordering, chunk geometry), plus whole-document `validate`/`is`/`parse`
+  trios combining structure and composition
+- [`zarr_metadata.builder`](builder.md) — validated construction:
+  one-shot `create_*` factories, one per document type
 - [`zarr_metadata.pydantic`](pydantic.md) — optional Pydantic field types
   over the models
 - [`zarr_metadata.v2`](v2.md) — `TypedDict` shapes for Zarr v2 documents

packages/zarr-metadata/docs/api/rules.md

@@ -0,0 +1,5 @@
+---
+title: rules
+---
+
+::: zarr_metadata.rules

packages/zarr-metadata/docs/index.md

@@ -71,11 +71,19 @@ members that the strict model parser rejects.
 
 The model validators enforce the declared document structure and a small set
 of context-free consistency rules, including fixed format literals, finite
-JSON numbers, non-negative dimensions, non-empty v3 codec pipelines, and one
-`dimension_names` entry per array dimension. They do not interpret extension
-names or configurations, resolve codec pipelines, or decide whether a data
-type, chunk grid, codec, or storage transformer is supported. Those decisions
-belong to consumer implementations.
+JSON numbers, non-negative dimensions, and non-empty v3 codec pipelines.
+They do not interpret extension names or configurations.
+
+The composition rules (`zarr_metadata.rules`) judge the document as a whole:
+fill values against data types, codec pipeline ordering, chunk-grid
+geometry against `shape`, one `dimension_names` entry per array dimension,
+and the canonical configuration shapes of the codecs, chunk grids, chunk
+key encodings, and data types this package defines. Unknown extension names
+are left unjudged. The rules model canonical documents and are deliberately
+stricter than any given implementation: an implementation may coerce
+ambiguous input as it sees fit and then validate the canonical result.
+Nothing here decides whether a data type, chunk grid, codec, or storage
+transformer is *supported*; that belongs to consumer implementations.
 
 ## Scope
 

packages/zarr-metadata/mkdocs.yml

@@ -16,6 +16,8 @@ nav:
   - API Reference:
       - api/index.md
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr_metadata.model</code>': api/model.md
+      - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr_metadata.rules</code>': api/rules.md
+      - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr_metadata.builder</code>': api/builder.md
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr_metadata.pydantic</code>': api/pydantic.md
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr_metadata.v2</code>': api/v2.md
       - '<code class="doc-symbol doc-symbol-toc doc-symbol-module"></code> <code>zarr_metadata.v3</code>':

packages/zarr-metadata/pyproject.toml

@@ -47,7 +47,7 @@ Changelog = "https://github.com/zarr-developers/zarr-python/blob/main/packages/z
 Documentation = "https://zarr-metadata.readthedocs.io/"
 
 [dependency-groups]
-test = ["pytest", "pydantic>=2.13", "jsonschema"]
+test = ["pytest", "pydantic>=2.13", "jsonschema", "hypothesis"]
 docs = [
     # Pins match the zarr-python docs environment in the repo-root
     # pyproject.toml so the two sites render with the same toolchain.

packages/zarr-metadata/src/zarr_metadata/builder/__init__.py

@@ -0,0 +1,32 @@
+"""Validated construction of Zarr metadata documents.
+
+`create_*` factories provide typed one-shot construction for every
+document TypedDict: required keys and value types are checked statically
+at literal-keyword call sites, and the runtime pass normalizes the input
+and applies structural and composition validation, raising one
+`MetadataValidationError` carrying every problem.
+
+Use `zarr_metadata.rules` to validate documents read from storage.
+"""
+
+from zarr_metadata.builder._create import (
+    create_zarr_v2_array_metadata_json,
+    create_zarr_v2_consolidated_metadata_json,
+    create_zarr_v2_group_metadata_json,
+    create_zarr_v2_zarray_json,
+    create_zarr_v2_zgroup_json,
+    create_zarr_v3_array_metadata_json,
+    create_zarr_v3_consolidated_metadata_json,
+    create_zarr_v3_group_metadata_json,
+)
+
+__all__ = [
+    "create_zarr_v2_array_metadata_json",
+    "create_zarr_v2_consolidated_metadata_json",
+    "create_zarr_v2_group_metadata_json",
+    "create_zarr_v2_zarray_json",
+    "create_zarr_v2_zgroup_json",
+    "create_zarr_v3_array_metadata_json",
+    "create_zarr_v3_consolidated_metadata_json",
+    "create_zarr_v3_group_metadata_json",
+]