diff --git a/packages/zarr-metadata/README.md b/packages/zarr-metadata/README.md index 6b6b172aec..0a0aa89ec1 100644 --- a/packages/zarr-metadata/README.md +++ b/packages/zarr-metadata/README.md @@ -24,21 +24,27 @@ Two layers and an optional integration: ## What this is for The public `TypedDict` definitions describe the static JSON shape of Zarr -metadata. For strict, loc-aware validation of JSON loaded from disk, use the -model parser: +metadata. To judge JSON loaded from disk, structure and composition together, +use the rules layer; to get a normalized document model, use the model parser: ```python import json from zarr_metadata.model import ZarrV3ArrayMetadata +from zarr_metadata.rules import parse_array_metadata_v3 with open("zarr.json", "rb") as f: raw = json.load(f) -metadata = ZarrV3ArrayMetadata.from_json(raw) +document = parse_array_metadata_v3(raw) # raises with every problem found +metadata = ZarrV3ArrayMetadata.from_json(document) ``` -The optional Pydantic integration delegates raw input to the same strict -parser and returns the same normalized model class: +To construct a document, the `create_*` factories in `zarr_metadata.builder` +apply the same judgment to keyword arguments typed by the document's +`TypedDict`. + +The optional Pydantic integration runs raw input through the rules layer +and returns the same normalized model class: ```python from pydantic import TypeAdapter @@ -56,11 +62,19 @@ members that the strict model parser rejects. The model validators enforce the declared document structure and a small set of context-free consistency rules, including fixed format literals, finite -JSON numbers, non-negative dimensions, non-empty v3 codec pipelines, and one -`dimension_names` entry per array dimension. They do not interpret extension -names or configurations, resolve codec pipelines, or decide whether a data -type, chunk grid, codec, or storage transformer is supported. Those decisions -belong to consumer implementations. +JSON numbers, non-negative dimensions, and non-empty v3 codec pipelines. +They do not interpret extension names or configurations. + +The composition rules (`zarr_metadata.rules`) judge the document as a whole: +fill values against data types, codec pipeline ordering, chunk-grid +geometry against `shape`, one `dimension_names` entry per array dimension, +and the canonical configuration shapes of the codecs, chunk grids, chunk +key encodings, and data types this package defines. Unknown extension names +are left unjudged. The rules model canonical documents and are deliberately +stricter than any given implementation: an implementation may coerce +ambiguous input as it sees fit and then validate the canonical result. +Nothing here decides whether a data type, chunk grid, codec, or storage +transformer is *supported*; that belongs to consumer implementations. The Pydantic integration's generated JSON Schemas express independently checkable document structure and field constraints, but they are not a @@ -68,7 +82,8 @@ replacement for runtime model validation. Standard JSON Schema treats a mathematically integral number such as `1.0` as an integer, while the runtime boundary requires Python `int` values, and it cannot express arbitrary same-length relations such as `dimension_names` versus `shape` or v2 `chunks` -versus `shape`. Consumers should run the model parser after schema validation. +versus `shape`. Consumers should run the runtime validators after schema +validation. ## Scope diff --git a/packages/zarr-metadata/changes/318.feature.3.md b/packages/zarr-metadata/changes/318.feature.3.md new file mode 100644 index 0000000000..f699ac068f --- /dev/null +++ b/packages/zarr-metadata/changes/318.feature.3.md @@ -0,0 +1,14 @@ +Added `create_*` factories in `zarr_metadata.builder`, one per public +document TypedDict (`create_zarr_v3_array_metadata_json`, +`create_zarr_v3_group_metadata_json`, `create_zarr_v3_consolidated_metadata_json`, +`create_zarr_v2_array_metadata_json`, `create_zarr_v2_group_metadata_json`, +`create_zarr_v2_zarray_json`, `create_zarr_v2_zgroup_json`, +`create_zarr_v2_consolidated_metadata_json`), each taking +`**kwargs: Unpack[]`. Each factory copies and normalizes its +input, runs structural and composition validation, and raises one +`MetadataValidationError` containing all problems. The strict on-disk +`.zarray`/`.zgroup` factories reject `attributes` at runtime, and the v2 +consolidated factory validates each entry against the document shape its +path suffix selects. The open v3 array/group factories take an +`extensions=` mapping for extension fields (for type checkers without PEP +728 support) and reject names that shadow standard fields. diff --git a/packages/zarr-metadata/changes/318.feature.5.md b/packages/zarr-metadata/changes/318.feature.5.md new file mode 100644 index 0000000000..1a5433c683 --- /dev/null +++ b/packages/zarr-metadata/changes/318.feature.5.md @@ -0,0 +1,17 @@ +Added `check_*` entry points in `zarr_metadata.rules` returning a +discriminated `Valid[T] | Invalid`, for callers who want a document and +its problems in one value. + +The literal `valid` field narrows to either the normalized document or a +nonempty problem tuple: + +```python +result = check_array_metadata_v3(loaded) +if result.valid: + store(result.document) # typed ZarrV3ArrayMetadataJSON +else: + report(result.problems) # non-empty tuple of problems +``` + +Use `validate_*` to collect problems and `parse_*` to raise on invalid +input. diff --git a/packages/zarr-metadata/changes/318.feature.6.md b/packages/zarr-metadata/changes/318.feature.6.md new file mode 100644 index 0000000000..fe9d67eced --- /dev/null +++ b/packages/zarr-metadata/changes/318.feature.6.md @@ -0,0 +1,13 @@ +Unknown members inside a *known* entity's `configuration` (e.g. an extra +key in a `blosc` configuration) now report as their own `unknown_key` +problem kind, and no longer suppress the other rules about that entity. + +Whether configurations are closed remains unspecified +([zarr-specs#270](https://github.com/zarr-developers/zarr-specs/issues/270)), +so this package retains its strict reading with two safeguards: + +- callers can filter the dedicated `unknown_key` kind; +- unknown keys do not suppress other rules for the same entity. + +Model round-trips preserve unmodeled members. Shape-exact `TypeIs` guards +still reject them because the corresponding TypedDicts are closed. diff --git a/packages/zarr-metadata/changes/318.feature.md b/packages/zarr-metadata/changes/318.feature.md new file mode 100644 index 0000000000..0e8c5b0e41 --- /dev/null +++ b/packages/zarr-metadata/changes/318.feature.md @@ -0,0 +1,51 @@ +Added `zarr_metadata.rules`: composition rules for full metadata +documents. The package now models metadata in three layers with one +contract each — `model` checks structure element by element, `rules` +judges composition across the document, and `builder` constructs while +applying both. Rules are registered where they are defined; rules about a +particular codec, chunk grid, or data type live with that entity under +`rules._entities` and are dispatched by name, so adding an entity adds a +module there and changes nothing else. + +- **Rule sets**: `ZARR_V3_ARRAY_RULES` covers fill value vs. data type, + codec pipeline kind ordering, known-name shapes, + dimension-name counts, chunk-grid values (positive extents) and + geometry (regular rank; rectilinear rank and per-dimension chunk-size + sums, RLE pairs included), transpose orders (self-permutation at any + depth, rank agreement with `shape`), and sharding (inner `codecs` and + `index_codecs` judged as pipelines recursively at every nesting depth; + inner chunk shapes positive, rank-matched, and evenly dividing the + enclosing chunk, recursively). New `ZARR_V2_ARRAY_RULES` + (chunks/shape rank agreement) and `ZARR_V3_GROUP_RULES` (inline + consolidated metadata recurses, judging each embedded child document + by its own rules at its path). +- **Read-side trios**: `validate_*` / `is_*` / `parse_*` for array and + group documents in both format versions mirror the model layer's + grammar with a stronger judgment — structure *and* composition, every + problem reported together, JSON arrays normalized to tuples before + judgment. The `is_*` functions deliberately return `bool` rather than + `TypeIs`: a composition-invalid document is still an instance of the + TypedDict, so only the structural layer can narrow honestly. +- **Boundary change**: two composition checks that lived in the + structural validator moved here — v3 `dimension_names` vs `shape` and + v2 `chunks` vs `shape` rank agreement. `zarr_metadata.model`'s + validators, parsers, and dataclasses now accept those documents (they + are lossless, structurally well-formed representations of what a store + may contain); use the `rules` trios to judge them. This also removes + the double report the overlap used to produce. +- **Strictness stance**, now documented on the package: `zarr_metadata` + models canonical documents and is deliberately stricter than any given + implementation; implementations coerce ambiguous input as they see fit + and then validate the canonical result. + +- **Codec chains are judged against the array each codec receives**: + `transpose` permutes the shape and `cast_value` changes the data type + seen by everything after it, so a shard behind a transpose must divide + the transposed chunk, and a `bytes` codec behind a cast needs an + endianness for the *target* type. `zarr_metadata.v3.codec.kind` sorts + known codec names into the spec's three pipeline kinds. +- **Pydantic field types** for array and group documents now run the + composition rules as well as structural validation. + +Known follow-up: v2 fill-value/dtype consistency (NumPy dtype grammar) +has no rule yet. diff --git a/packages/zarr-metadata/docs/api/builder.md b/packages/zarr-metadata/docs/api/builder.md new file mode 100644 index 0000000000..50bf9ac278 --- /dev/null +++ b/packages/zarr-metadata/docs/api/builder.md @@ -0,0 +1,5 @@ +--- +title: builder +--- + +::: zarr_metadata.builder diff --git a/packages/zarr-metadata/docs/api/index.md b/packages/zarr-metadata/docs/api/index.md index 5e230c7aa2..1a73ec3842 100644 --- a/packages/zarr-metadata/docs/api/index.md +++ b/packages/zarr-metadata/docs/api/index.md @@ -8,6 +8,12 @@ The package is organized to mirror the structure of the Zarr specifications: - [`zarr_metadata.model`](model.md) — frozen-dataclass document models, structural validators, loc-aware parsers, and the `UNSET` sentinel +- [`zarr_metadata.rules`](rules.md) — composition rules: cross-field + judgments over full documents (fill value vs. data type, codec pipeline + ordering, chunk geometry), plus whole-document `validate`/`is`/`parse` + trios combining structure and composition +- [`zarr_metadata.builder`](builder.md) — validated construction: + one-shot `create_*` factories, one per document type - [`zarr_metadata.pydantic`](pydantic.md) — optional Pydantic field types over the models - [`zarr_metadata.v2`](v2.md) — `TypedDict` shapes for Zarr v2 documents diff --git a/packages/zarr-metadata/docs/api/rules.md b/packages/zarr-metadata/docs/api/rules.md new file mode 100644 index 0000000000..ef010b72e2 --- /dev/null +++ b/packages/zarr-metadata/docs/api/rules.md @@ -0,0 +1,5 @@ +--- +title: rules +--- + +::: zarr_metadata.rules diff --git a/packages/zarr-metadata/docs/index.md b/packages/zarr-metadata/docs/index.md index 58e4f290c6..ee6514005a 100644 --- a/packages/zarr-metadata/docs/index.md +++ b/packages/zarr-metadata/docs/index.md @@ -71,11 +71,19 @@ members that the strict model parser rejects. The model validators enforce the declared document structure and a small set of context-free consistency rules, including fixed format literals, finite -JSON numbers, non-negative dimensions, non-empty v3 codec pipelines, and one -`dimension_names` entry per array dimension. They do not interpret extension -names or configurations, resolve codec pipelines, or decide whether a data -type, chunk grid, codec, or storage transformer is supported. Those decisions -belong to consumer implementations. +JSON numbers, non-negative dimensions, and non-empty v3 codec pipelines. +They do not interpret extension names or configurations. + +The composition rules (`zarr_metadata.rules`) judge the document as a whole: +fill values against data types, codec pipeline ordering, chunk-grid +geometry against `shape`, one `dimension_names` entry per array dimension, +and the canonical configuration shapes of the codecs, chunk grids, chunk +key encodings, and data types this package defines. Unknown extension names +are left unjudged. The rules model canonical documents and are deliberately +stricter than any given implementation: an implementation may coerce +ambiguous input as it sees fit and then validate the canonical result. +Nothing here decides whether a data type, chunk grid, codec, or storage +transformer is *supported*; that belongs to consumer implementations. ## Scope diff --git a/packages/zarr-metadata/mkdocs.yml b/packages/zarr-metadata/mkdocs.yml index 18e1fc8c35..c7af51bcc9 100644 --- a/packages/zarr-metadata/mkdocs.yml +++ b/packages/zarr-metadata/mkdocs.yml @@ -16,6 +16,8 @@ nav: - API Reference: - api/index.md - ' zarr_metadata.model': api/model.md + - ' zarr_metadata.rules': api/rules.md + - ' zarr_metadata.builder': api/builder.md - ' zarr_metadata.pydantic': api/pydantic.md - ' zarr_metadata.v2': api/v2.md - ' zarr_metadata.v3': diff --git a/packages/zarr-metadata/pyproject.toml b/packages/zarr-metadata/pyproject.toml index f38710c860..081f2b3c48 100644 --- a/packages/zarr-metadata/pyproject.toml +++ b/packages/zarr-metadata/pyproject.toml @@ -47,7 +47,7 @@ Changelog = "https://github.com/zarr-developers/zarr-python/blob/main/packages/z Documentation = "https://zarr-metadata.readthedocs.io/" [dependency-groups] -test = ["pytest", "pydantic>=2.13", "jsonschema"] +test = ["pytest", "pydantic>=2.13", "jsonschema", "hypothesis"] docs = [ # Pins match the zarr-python docs environment in the repo-root # pyproject.toml so the two sites render with the same toolchain. diff --git a/packages/zarr-metadata/src/zarr_metadata/builder/__init__.py b/packages/zarr-metadata/src/zarr_metadata/builder/__init__.py new file mode 100644 index 0000000000..10dcb0dfe9 --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/builder/__init__.py @@ -0,0 +1,32 @@ +"""Validated construction of Zarr metadata documents. + +`create_*` factories provide typed one-shot construction for every +document TypedDict: required keys and value types are checked statically +at literal-keyword call sites, and the runtime pass normalizes the input +and applies structural and composition validation, raising one +`MetadataValidationError` carrying every problem. + +Use `zarr_metadata.rules` to validate documents read from storage. +""" + +from zarr_metadata.builder._create import ( + create_zarr_v2_array_metadata_json, + create_zarr_v2_consolidated_metadata_json, + create_zarr_v2_group_metadata_json, + create_zarr_v2_zarray_json, + create_zarr_v2_zgroup_json, + create_zarr_v3_array_metadata_json, + create_zarr_v3_consolidated_metadata_json, + create_zarr_v3_group_metadata_json, +) + +__all__ = [ + "create_zarr_v2_array_metadata_json", + "create_zarr_v2_consolidated_metadata_json", + "create_zarr_v2_group_metadata_json", + "create_zarr_v2_zarray_json", + "create_zarr_v2_zgroup_json", + "create_zarr_v3_array_metadata_json", + "create_zarr_v3_consolidated_metadata_json", + "create_zarr_v3_group_metadata_json", +] diff --git a/packages/zarr-metadata/src/zarr_metadata/builder/_create.py b/packages/zarr-metadata/src/zarr_metadata/builder/_create.py new file mode 100644 index 0000000000..58aac4c47b --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/builder/_create.py @@ -0,0 +1,333 @@ +"""One-shot factories for metadata document TypedDicts. + +Each factory copies and normalizes its input, then applies structural and +composition validation. Invalid input raises one +`MetadataValidationError`. V3 array and group factories accept extension +fields through `extensions=` for compatibility with type checkers that do +not support PEP 728. + +""" + +from __future__ import annotations + +import copy +from collections.abc import Mapping, Sequence +from typing import TYPE_CHECKING, cast + +from typing_extensions import Unpack + +from zarr_metadata.model._group import ZarrV2ConsolidatedMetadata +from zarr_metadata.model._validation import ( + ARRAY_METADATA_STANDARD_KEYS_V3, + GROUP_METADATA_STANDARD_KEYS_V3, + MetadataValidationError, + ValidationProblem, + arrays_to_tuples, + parse_array_metadata_v2, + parse_array_metadata_v3, + parse_group_metadata_v2, + parse_group_metadata_v3, + validate_consolidated_metadata_v3, +) +from zarr_metadata.rules import ( + ZARR_V2_ARRAY_RULES, + ZARR_V3_ARRAY_RULES, + ZARR_V3_GROUP_RULES, + run_rules, + validate_array_metadata_v2, + validate_group_metadata_v2, +) +from zarr_metadata.rules._v3_group import consolidated_entries_problems + +if TYPE_CHECKING: + from collections.abc import Set as AbstractSet + + from zarr_metadata.v2.array import ZarrV2ArrayMetadataJSON, ZarrV2ZArrayJSON + from zarr_metadata.v2.consolidated import ZarrV2ConsolidatedMetadataJSON + from zarr_metadata.v2.group import ZarrV2GroupMetadataJSON, ZarrV2ZGroupJSON + from zarr_metadata.v3.array import ZarrV3ArrayMetadataJSON, ZarrV3ExtensionField + from zarr_metadata.v3.consolidated import ZarrV3ConsolidatedMetadataJSON + from zarr_metadata.v3.group import ZarrV3GroupMetadataJSON + + +def _merged_with_extensions( + kwargs: Mapping[str, object], + extensions: Mapping[str, ZarrV3ExtensionField] | None, + standard_keys: AbstractSet[str], +) -> tuple[dict[str, object], list[ValidationProblem]]: + """`kwargs` plus `extensions`, refusing extension names that shadow standard keys. + + A colliding name is reported and *not* merged, so a later structural or + semantic pass judges the standard field the caller actually passed + rather than a value smuggled in through the extension hatch. + """ + document = dict(kwargs) + problems: list[ValidationProblem] = [] + for name, value in (extensions or {}).items(): + if name in standard_keys: + problems.append( + ValidationProblem( + (name,), + f"{name!r} is a standard metadata key; pass it as a keyword argument", + "invalid_value", + ) + ) + else: + document[name] = value + return document, problems + + +def _normalized(document: Mapping[str, object]) -> dict[str, object]: + """A deep copy of `document` with JSON arrays materialized as tuples. + + Deep-copying first means the returned document shares no mutable state + with the caller's arguments: mutating an input after the factory + returns cannot alter the validated result. + """ + return cast("dict[str, object]", arrays_to_tuples(copy.deepcopy(dict(document)))) + + +def _raise_if_problems(problems: Sequence[ValidationProblem]) -> None: + if len(problems) != 0: + raise MetadataValidationError(problems) + + +def _reject_attributes(document: Mapping[str, object]) -> tuple[ValidationProblem, ...]: + """Problems for an `attributes` key in a strict on-disk v2 document. + + The strict `.zarray` / `.zgroup` shapes exclude `attributes` (it lives + in the sibling `.zattrs` file). The signature enforces that statically + at keyword call sites; this is the runtime backstop for `**`-splatted + and untyped callers, without which the merged-form parser would accept + the key and the returned value would not be the type it claims. + """ + if "attributes" not in document: + return () + return ( + ValidationProblem( + ("attributes",), + "'attributes' is not part of the on-disk document (it belongs to the " + "sibling .zattrs file); use the merged-form factory instead", + "invalid_value", + ), + ) + + +def create_zarr_v3_array_metadata_json( + *, + extensions: Mapping[str, ZarrV3ExtensionField] | None = None, + **kwargs: Unpack[ZarrV3ArrayMetadataJSON], +) -> ZarrV3ArrayMetadataJSON: + """A validated v3 array metadata document (the `zarr.json` content for an array). + + Required keys are enforced statically by the signature; at runtime the + document is checked structurally (via the model layer's parser) and + semantically (via `ZARR_V3_ARRAY_RULES`), and every problem from both + passes is raised together in one `MetadataValidationError`. Extension + fields go in `extensions`; names that shadow standard keys are rejected. + """ + document, problems = _merged_with_extensions( + kwargs, extensions, ARRAY_METADATA_STANDARD_KEYS_V3 + ) + normalized = _normalized(document) + parsed: ZarrV3ArrayMetadataJSON | None = None + try: + parsed = parse_array_metadata_v3(normalized) + except MetadataValidationError as error: + problems.extend(error.problems) + problems.extend(run_rules(ZARR_V3_ARRAY_RULES, normalized)) + _raise_if_problems(problems) + assert parsed is not None + return parsed + + +def create_zarr_v3_group_metadata_json( + *, + extensions: Mapping[str, ZarrV3ExtensionField] | None = None, + **kwargs: Unpack[ZarrV3GroupMetadataJSON], +) -> ZarrV3GroupMetadataJSON: + """A validated v3 group metadata document (the `zarr.json` content for a group). + + Extension fields go in `extensions`; names that shadow standard keys + are rejected. The composition rules recurse into inline consolidated + metadata, so an embedded child document invalid under its own rules + is reported here, at its path. + """ + document, problems = _merged_with_extensions( + kwargs, extensions, GROUP_METADATA_STANDARD_KEYS_V3 + ) + normalized = _normalized(document) + parsed: ZarrV3GroupMetadataJSON | None = None + try: + parsed = parse_group_metadata_v3(normalized) + except MetadataValidationError as error: + problems.extend(error.problems) + problems.extend(run_rules(ZARR_V3_GROUP_RULES, normalized)) + _raise_if_problems(problems) + assert parsed is not None + return parsed + + +def create_zarr_v3_consolidated_metadata_json( + **kwargs: Unpack[ZarrV3ConsolidatedMetadataJSON], +) -> ZarrV3ConsolidatedMetadataJSON: + """A validated v3 inline consolidated metadata object. + + This is the value embedded in a v3 group document under the + `consolidated_metadata` key, not a store document of its own. + """ + normalized = _normalized(kwargs) + _raise_if_problems( + validate_consolidated_metadata_v3(normalized) + consolidated_entries_problems(normalized) + ) + return cast("ZarrV3ConsolidatedMetadataJSON", normalized) + + +def create_zarr_v2_array_metadata_json( + **kwargs: Unpack[ZarrV2ArrayMetadataJSON], +) -> ZarrV2ArrayMetadataJSON: + """A validated v2 array metadata document, in-memory merged form. + + Models `.zarray` plus the sibling `.zattrs` folded in as `attributes`. + For the strict on-disk `.zarray` shape use `create_zarr_v2_zarray_json`. + """ + normalized = _normalized(kwargs) + parsed: ZarrV2ArrayMetadataJSON | None = None + problems: list[ValidationProblem] = [] + try: + parsed = parse_array_metadata_v2(normalized) + except MetadataValidationError as error: + problems.extend(error.problems) + problems.extend(run_rules(ZARR_V2_ARRAY_RULES, normalized)) + _raise_if_problems(problems) + assert parsed is not None + return parsed + + +def create_zarr_v2_group_metadata_json( + **kwargs: Unpack[ZarrV2GroupMetadataJSON], +) -> ZarrV2GroupMetadataJSON: + """A validated v2 group metadata document, in-memory merged form. + + Models `.zgroup` plus the sibling `.zattrs` folded in as `attributes`. + For the strict on-disk `.zgroup` shape use `create_zarr_v2_zgroup_json`. + """ + parsed: ZarrV2GroupMetadataJSON | None = None + problems: list[ValidationProblem] = [] + try: + parsed = parse_group_metadata_v2(_normalized(kwargs)) + except MetadataValidationError as error: + problems.extend(error.problems) + _raise_if_problems(problems) + assert parsed is not None + return parsed + + +def create_zarr_v2_zarray_json(**kwargs: Unpack[ZarrV2ZArrayJSON]) -> ZarrV2ZArrayJSON: + """A validated on-disk `.zarray` document (strict form, no `attributes`). + + Structurally checked with the merged-form parser plus a runtime + rejection of `attributes`: the strict shape is the merged shape minus + `attributes`, and the runtime check holds for callers the signature's + static exclusion cannot see (`**`-splatted mappings, untyped code). + """ + normalized = _normalized(kwargs) + problems: list[ValidationProblem] = list(_reject_attributes(normalized)) + parsed: ZarrV2ArrayMetadataJSON | None = None + try: + parsed = parse_array_metadata_v2(normalized) + except MetadataValidationError as error: + problems.extend(error.problems) + problems.extend(run_rules(ZARR_V2_ARRAY_RULES, normalized)) + _raise_if_problems(problems) + assert parsed is not None + return cast("ZarrV2ZArrayJSON", parsed) + + +def create_zarr_v2_zgroup_json(**kwargs: Unpack[ZarrV2ZGroupJSON]) -> ZarrV2ZGroupJSON: + """A validated on-disk `.zgroup` document (strict form, no `attributes`). + + Structurally checked with the merged-form parser plus a runtime + rejection of `attributes`: the strict shape is the merged shape minus + `attributes`, and the runtime check holds for callers the signature's + static exclusion cannot see (`**`-splatted mappings, untyped code). + """ + normalized = _normalized(kwargs) + problems: list[ValidationProblem] = list(_reject_attributes(normalized)) + parsed: ZarrV2GroupMetadataJSON | None = None + try: + parsed = parse_group_metadata_v2(normalized) + except MetadataValidationError as error: + problems.extend(error.problems) + _raise_if_problems(problems) + assert parsed is not None + return cast("ZarrV2ZGroupJSON", parsed) + + +def _validate_v2_consolidated_envelope( + document: Mapping[str, object], +) -> tuple[ValidationProblem, ...]: + """Every reason `document` is not a `.zmetadata` envelope. + + The envelope itself is the model layer's judgment; on top of it, each + entry's path suffix selects the strict on-disk document shape that its + value must satisfy. + """ + try: + ZarrV2ConsolidatedMetadata.from_json(document) + except MetadataValidationError as error: + return error.problems + problems: list[ValidationProblem] = [] + for key, entry in cast("Mapping[str, object]", document["metadata"]).items(): + if not isinstance(entry, Mapping): + problems.append( + ValidationProblem(("metadata", key), "expected a JSON object", "invalid_type") + ) + continue + entry_mapping = cast("Mapping[str, object]", entry) + if key.endswith(".zarray"): + nested = validate_array_metadata_v2(entry_mapping) + _reject_attributes(entry_mapping) + elif key.endswith(".zgroup"): + nested = validate_group_metadata_v2(entry_mapping) + _reject_attributes(entry_mapping) + elif key.endswith(".zattrs"): + nested = () + else: + nested = ( + ValidationProblem( + (), + "expected a v2 metadata file suffix: .zarray, .zgroup, or .zattrs", + "invalid_value", + ), + ) + problems.extend( + ValidationProblem(("metadata", key, *found.loc), found.message, found.kind) + for found in nested + ) + return tuple(problems) + + +def create_zarr_v2_consolidated_metadata_json( + **kwargs: Unpack[ZarrV2ConsolidatedMetadataJSON], +) -> ZarrV2ConsolidatedMetadataJSON: + """A validated `.zmetadata` consolidated metadata document. + + The runtime pass checks the envelope and validates each nested value + against the strict document shape selected by its path suffix. This is + the runtime backstop for callers the signature's static enforcement + cannot see (`**`-splatted mappings, untyped code). + """ + normalized = _normalized(kwargs) + _raise_if_problems(_validate_v2_consolidated_envelope(normalized)) + return cast("ZarrV2ConsolidatedMetadataJSON", normalized) + + +__all__ = [ + "create_zarr_v2_array_metadata_json", + "create_zarr_v2_consolidated_metadata_json", + "create_zarr_v2_group_metadata_json", + "create_zarr_v2_zarray_json", + "create_zarr_v2_zgroup_json", + "create_zarr_v3_array_metadata_json", + "create_zarr_v3_consolidated_metadata_json", + "create_zarr_v3_group_metadata_json", +] diff --git a/packages/zarr-metadata/src/zarr_metadata/model/_validation.py b/packages/zarr-metadata/src/zarr_metadata/model/_validation.py index b4245a8e3d..c7ee219a6d 100644 --- a/packages/zarr-metadata/src/zarr_metadata/model/_validation.py +++ b/packages/zarr-metadata/src/zarr_metadata/model/_validation.py @@ -7,8 +7,8 @@ Every `ValidationProblem` carries a machine-readable `kind` alongside its human-readable `message`, so consumers can dispatch on the failure mode -(`missing_key`, `invalid_type`, `invalid_value`, `invalid_json`) without -string-matching messages. +(`missing_key`, `invalid_type`, `invalid_value`, `invalid_json`, +`unknown_key`) without string-matching messages. """ from __future__ import annotations @@ -28,7 +28,7 @@ from zarr_metadata.v3.array import ZarrV3ArrayMetadataJSON from zarr_metadata.v3.group import ZarrV3GroupMetadataJSON -ProblemKind = Literal["missing_key", "invalid_type", "invalid_value", "invalid_json"] +ProblemKind = Literal["missing_key", "invalid_type", "invalid_value", "invalid_json", "unknown_key"] """Machine-readable classification of a `ValidationProblem`. - `missing_key`: a required key (document key or store key) is absent. @@ -37,6 +37,15 @@ - `invalid_value`: a value has an acceptable type but an invalid content (e.g. `zarr_format: 2` in a v3 document, `order: "Q"`). - `invalid_json`: bytes that do not decode as JSON. +- `unknown_key`: a member this package does not model appears inside an + entity whose shape it does model (e.g. an extra key in a `blosc` + configuration). Distinguished from `invalid_value` because the Zarr v3 + spec does not say whether a `configuration` is closed + (zarr-developers/zarr-specs#270 has been open since 2023), so this is + the package's strict reading rather than a definite violation: a + document carrying one is very likely fine, just written by something + that models more than we do. Callers that prefer tolerance can filter + this kind out; the package itself never lets it mask other findings. """ @@ -550,16 +559,8 @@ def validate_array_metadata_v3(value: object) -> tuple[ValidationProblem, ...]: ("dimension_names",), "expected items of str or None", "invalid_type" ) ) - elif _is_int_sequence(doc.get("shape")) and len(cast("Sequence[object]", names)) != len( - cast("Sequence[int]", doc["shape"]) - ): - problems.append( - ValidationProblem( - ("dimension_names",), - "expected one name per dimension of shape", - "invalid_value", - ) - ) + # Whether the names count matches shape's dimensionality is a + # composition judgment, owned by zarr_metadata.rules. return tuple(problems) @@ -597,26 +598,10 @@ def validate_array_metadata_v2(value: object) -> tuple[ValidationProblem, ...]: _unexpected_keys(ARRAY_METADATA_STANDARD_KEYS_V2, cast("Mapping[object, object]", value)) ) problems.extend(_check_literal(doc, "zarr_format", 2)) - shape_problems = _validate_dim_sequence(doc, "shape") - chunks_problems = _validate_dim_sequence(doc, "chunks") - problems.extend(shape_problems) - problems.extend(chunks_problems) - if ( - len(shape_problems) == 0 - and len(chunks_problems) == 0 - and _is_int_sequence(doc.get("shape")) - and _is_int_sequence(doc.get("chunks")) - ): - shape = cast("Sequence[int]", doc["shape"]) - chunks = cast("Sequence[int]", doc["chunks"]) - if len(shape) != len(chunks): - problems.append( - ValidationProblem( - ("chunks",), - "expected the same number of dimensions as shape", - "invalid_value", - ) - ) + problems.extend(_validate_dim_sequence(doc, "shape")) + problems.extend(_validate_dim_sequence(doc, "chunks")) + # Whether chunks matches shape's dimensionality is a composition + # judgment, owned by zarr_metadata.rules. if "dtype" in doc and not _is_dtype_v2(doc["dtype"]): problems.append( ValidationProblem( diff --git a/packages/zarr-metadata/src/zarr_metadata/pydantic.py b/packages/zarr-metadata/src/zarr_metadata/pydantic.py index 5b0c9e5b57..11d27d0a7e 100644 --- a/packages/zarr-metadata/src/zarr_metadata/pydantic.py +++ b/packages/zarr-metadata/src/zarr_metadata/pydantic.py @@ -6,13 +6,17 @@ Each exported name is an `Annotated` field type over the corresponding core model class — the instances ARE the core classes, so values interoperate freely with non-pydantic code (equality, isinstance, nesting). Validation -delegates to the library: a raw document routes through `from_json` (the -single source of truth for structural validation and normalization, so -pydantic's field-level coercion can never bypass it), an existing model -instance passes through unchanged, and serialization emits the canonical -document via `to_json`. `MetadataValidationError` subclasses `ValueError`, -so a failed parse surfaces as a pydantic `ValidationError` carrying the -loc-annotated problem messages. +delegates to the library: a raw document is judged by `zarr_metadata.rules` +(structure and composition together) and then normalized through +`from_json`, so pydantic's field-level coercion can never bypass either +layer; an existing model instance passes through unchanged, and +serialization emits the canonical document via `to_json`. +`MetadataValidationError` subclasses `ValueError`, so a failed parse +surfaces as a pydantic `ValidationError` carrying the loc-annotated +problem messages. + +The v2 consolidated field type and the bare metadata-field type carry no +composition rules and are validated structurally. Usage: @@ -33,6 +37,7 @@ class ArrayManifest(BaseModel): from pydantic import BeforeValidator, InstanceOf, PlainSerializer from zarr_metadata import model as _model +from zarr_metadata import rules as _rules from zarr_metadata._pydantic_schema import ( ZarrV2ArrayMetadataJSON as _ZarrV2ArrayMetadataSchema, ) @@ -54,6 +59,8 @@ class ArrayManifest(BaseModel): from zarr_metadata._pydantic_schema import ( ZarrV3MetadataFieldJSON as _ZarrV3MetadataFieldSchema, ) +from zarr_metadata.model._validation import arrays_to_tuples, validate_consolidated_metadata_v3 +from zarr_metadata.rules._v3_group import consolidated_entries_problems if TYPE_CHECKING: from collections.abc import Callable @@ -72,10 +79,33 @@ def coerce(value: object) -> _M: return coerce +def _judged_by( + validate: Callable[[object], tuple[_model.ValidationProblem, ...]], + parse: Callable[[object], _M], +) -> Callable[[object], _M]: + """`parse`, preceded by a whole-document judgment that raises on any problem.""" + + def judged(value: object) -> _M: + problems = validate(value) + if len(problems) != 0: + raise _model.MetadataValidationError(problems) + return parse(value) + + return judged + + +def _validate_consolidated_v3(value: object) -> tuple[_model.ValidationProblem, ...]: + normalized = arrays_to_tuples(value) + return validate_consolidated_metadata_v3(normalized) + consolidated_entries_problems(normalized) + + ZarrV3ArrayMetadata = Annotated[ InstanceOf[_model.ZarrV3ArrayMetadata], BeforeValidator( - _coerce_to(_model.ZarrV3ArrayMetadata, _model.ZarrV3ArrayMetadata.from_json), + _coerce_to( + _model.ZarrV3ArrayMetadata, + _judged_by(_rules.validate_array_metadata_v3, _model.ZarrV3ArrayMetadata.from_json), + ), json_schema_input_type=_ZarrV3ArrayMetadataSchema, ), PlainSerializer(_model.ZarrV3ArrayMetadata.to_json, return_type=_ZarrV3ArrayMetadataSchema), @@ -85,7 +115,10 @@ def coerce(value: object) -> _M: ZarrV2ArrayMetadata = Annotated[ InstanceOf[_model.ZarrV2ArrayMetadata], BeforeValidator( - _coerce_to(_model.ZarrV2ArrayMetadata, _model.ZarrV2ArrayMetadata.from_json), + _coerce_to( + _model.ZarrV2ArrayMetadata, + _judged_by(_rules.validate_array_metadata_v2, _model.ZarrV2ArrayMetadata.from_json), + ), json_schema_input_type=_ZarrV2ArrayMetadataSchema, ), PlainSerializer(_model.ZarrV2ArrayMetadata.to_json, return_type=_ZarrV2ArrayMetadataSchema), @@ -95,7 +128,10 @@ def coerce(value: object) -> _M: ZarrV3GroupMetadata = Annotated[ InstanceOf[_model.ZarrV3GroupMetadata], BeforeValidator( - _coerce_to(_model.ZarrV3GroupMetadata, _model.ZarrV3GroupMetadata.from_json), + _coerce_to( + _model.ZarrV3GroupMetadata, + _judged_by(_rules.validate_group_metadata_v3, _model.ZarrV3GroupMetadata.from_json), + ), json_schema_input_type=_ZarrV3GroupMetadataSchema, ), PlainSerializer(_model.ZarrV3GroupMetadata.to_json, return_type=_ZarrV3GroupMetadataSchema), @@ -105,7 +141,10 @@ def coerce(value: object) -> _M: ZarrV2GroupMetadata = Annotated[ InstanceOf[_model.ZarrV2GroupMetadata], BeforeValidator( - _coerce_to(_model.ZarrV2GroupMetadata, _model.ZarrV2GroupMetadata.from_json), + _coerce_to( + _model.ZarrV2GroupMetadata, + _judged_by(_rules.validate_group_metadata_v2, _model.ZarrV2GroupMetadata.from_json), + ), json_schema_input_type=_ZarrV2GroupMetadataSchema, ), PlainSerializer(_model.ZarrV2GroupMetadata.to_json, return_type=_ZarrV2GroupMetadataSchema), @@ -117,7 +156,7 @@ def coerce(value: object) -> _M: BeforeValidator( _coerce_to( _model.ZarrV3ConsolidatedMetadata, - _model.ZarrV3ConsolidatedMetadata.from_json, + _judged_by(_validate_consolidated_v3, _model.ZarrV3ConsolidatedMetadata.from_json), ), json_schema_input_type=_ZarrV3ConsolidatedMetadataSchema, ), diff --git a/packages/zarr-metadata/src/zarr_metadata/rules/__init__.py b/packages/zarr-metadata/src/zarr_metadata/rules/__init__.py new file mode 100644 index 0000000000..0631f838c8 --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/rules/__init__.py @@ -0,0 +1,73 @@ +"""Validate structure and composition of Zarr metadata documents. + +`zarr_metadata.model` checks JSON structure. This module also checks +cross-field constraints such as fill-value compatibility, codec +ordering, and dimension counts. Its `validate_*`, `is_*`, and `parse_*` +functions mirror the model API; `check_*` returns `Valid[T] | Invalid`. + +Rules target canonical metadata and may be stricter than readers that +coerce inputs. Unknown entity names are left unjudged. Known entities +must match their modeled shape; extra configuration keys produce an +`unknown_key` problem without suppressing other checks. Model +round-trips preserve those unmodeled members. +""" + +from zarr_metadata.rules._documents import ( + is_array_metadata_v2, + is_array_metadata_v3, + is_group_metadata_v2, + is_group_metadata_v3, + parse_array_metadata_v2, + parse_array_metadata_v3, + parse_group_metadata_v2, + parse_group_metadata_v3, + validate_array_metadata_v2, + validate_array_metadata_v3, + validate_group_metadata_v2, + validate_group_metadata_v3, +) +from zarr_metadata.rules._engine import Rule, RuleCheck, applicable, run_rules +from zarr_metadata.rules._result import ( + Invalid, + Valid, + ValidationResult, + check_array_metadata_v2, + check_array_metadata_v3, + check_group_metadata_v2, + check_group_metadata_v3, +) +from zarr_metadata.rules._v2_array import ZARR_V2_ARRAY, ZARR_V2_ARRAY_RULES +from zarr_metadata.rules._v3_array import ZARR_V3_ARRAY, ZARR_V3_ARRAY_RULES +from zarr_metadata.rules._v3_group import ZARR_V3_GROUP, ZARR_V3_GROUP_RULES + +__all__ = [ + "ZARR_V2_ARRAY", + "ZARR_V2_ARRAY_RULES", + "ZARR_V3_ARRAY", + "ZARR_V3_ARRAY_RULES", + "ZARR_V3_GROUP", + "ZARR_V3_GROUP_RULES", + "Invalid", + "Rule", + "RuleCheck", + "Valid", + "ValidationResult", + "applicable", + "check_array_metadata_v2", + "check_array_metadata_v3", + "check_group_metadata_v2", + "check_group_metadata_v3", + "is_array_metadata_v2", + "is_array_metadata_v3", + "is_group_metadata_v2", + "is_group_metadata_v3", + "parse_array_metadata_v2", + "parse_array_metadata_v3", + "parse_group_metadata_v2", + "parse_group_metadata_v3", + "run_rules", + "validate_array_metadata_v2", + "validate_array_metadata_v3", + "validate_group_metadata_v2", + "validate_group_metadata_v3", +] diff --git a/packages/zarr-metadata/src/zarr_metadata/rules/_documents.py b/packages/zarr-metadata/src/zarr_metadata/rules/_documents.py new file mode 100644 index 0000000000..d88ff862d0 --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/rules/_documents.py @@ -0,0 +1,187 @@ +"""Whole-document structural and composition validation. + +These `validate_*`, `is_*`, and `parse_*` functions mirror the model API +but apply both validation layers. The `is_*` functions return `bool`, not +`TypeIs`: composition validity is stricter than TypedDict membership. +Use `zarr_metadata.model.is_*` for type narrowing. +""" + +from __future__ import annotations + +from collections.abc import Mapping +from typing import TYPE_CHECKING, cast + +from zarr_metadata.model._validation import ( + MetadataValidationError, + arrays_to_tuples, +) +from zarr_metadata.model._validation import ( + validate_array_metadata_v2 as _validate_structure_v2, +) +from zarr_metadata.model._validation import ( + validate_array_metadata_v3 as _validate_structure_v3, +) +from zarr_metadata.model._validation import ( + validate_group_metadata_v2 as _validate_group_structure_v2, +) +from zarr_metadata.model._validation import ( + validate_group_metadata_v3 as _validate_group_structure_v3, +) +from zarr_metadata.rules._engine import run_rules +from zarr_metadata.rules._v2_array import ZARR_V2_ARRAY_RULES +from zarr_metadata.rules._v3_array import ZARR_V3_ARRAY_RULES +from zarr_metadata.rules._v3_group import ZARR_V3_GROUP_RULES + +if TYPE_CHECKING: + from zarr_metadata.model._validation import ValidationProblem + from zarr_metadata.v2.array import ZarrV2ArrayMetadataJSON + from zarr_metadata.v2.group import ZarrV2GroupMetadataJSON + from zarr_metadata.v3.array import ZarrV3ArrayMetadataJSON + from zarr_metadata.v3.group import ZarrV3GroupMetadataJSON + + +def validate_array_metadata_v3(value: object) -> tuple[ValidationProblem, ...]: + """Every reason `value` is not a valid v3 array document. + + Structural problems (from the model layer) and composition problems + (from `ZARR_V3_ARRAY_RULES`) are reported together. JSON arrays are + normalized to tuples before judgment, so list-spelled documents + (e.g. fresh `json.loads` output) are judged at the canonical data + level rather than rejected for their spelling. + """ + normalized = arrays_to_tuples(value) + problems = _validate_structure_v3(normalized) + if isinstance(normalized, Mapping): + document = cast("Mapping[str, object]", normalized) + problems = problems + run_rules(ZARR_V3_ARRAY_RULES, document) + return tuple(problems) + + +def is_array_metadata_v3(value: object) -> bool: + """Whether `value` is a structurally and compositionally valid v3 array doc. + + Deliberately not a `TypeIs` guard — see the module docstring. Use + `zarr_metadata.model.is_array_metadata_v3` to narrow. + """ + return len(validate_array_metadata_v3(value)) == 0 + + +def parse_array_metadata_v3(value: object) -> ZarrV3ArrayMetadataJSON: + """Return `value` as a valid `ZarrV3ArrayMetadataJSON`, or raise. + + Normalizes JSON arrays to tuples, then raises a single + `MetadataValidationError` carrying every structural and composition + problem found. + """ + normalized = arrays_to_tuples(value) + problems = validate_array_metadata_v3(normalized) + if len(problems) != 0: + raise MetadataValidationError(problems) + return cast("ZarrV3ArrayMetadataJSON", normalized) + + +def validate_array_metadata_v2(value: object) -> tuple[ValidationProblem, ...]: + """Every reason `value` is not a valid v2 array document (merged form). + + JSON arrays are normalized to tuples before judgment, as in + `validate_array_metadata_v3`. + """ + normalized = arrays_to_tuples(value) + problems = _validate_structure_v2(normalized) + if isinstance(normalized, Mapping): + document = cast("Mapping[str, object]", normalized) + problems = problems + run_rules(ZARR_V2_ARRAY_RULES, document) + return tuple(problems) + + +def is_array_metadata_v2(value: object) -> bool: + """Whether `value` is a structurally and compositionally valid v2 array doc. + + Deliberately not a `TypeIs` guard — see the module docstring. + """ + return len(validate_array_metadata_v2(value)) == 0 + + +def parse_array_metadata_v2(value: object) -> ZarrV2ArrayMetadataJSON: + """Return `value` as a valid `ZarrV2ArrayMetadataJSON`, or raise. + + Normalizes JSON arrays to tuples, then raises a single + `MetadataValidationError` carrying every structural and composition + problem found. + """ + normalized = arrays_to_tuples(value) + problems = validate_array_metadata_v2(normalized) + if len(problems) != 0: + raise MetadataValidationError(problems) + return cast("ZarrV2ArrayMetadataJSON", normalized) + + +def validate_group_metadata_v3(value: object) -> tuple[ValidationProblem, ...]: + """Every reason `value` is not a valid v3 group document. + + Composition rules recurse into inline consolidated metadata, so a + consolidated child document invalid under its own rules is reported + here, at its path. + """ + normalized = arrays_to_tuples(value) + problems = _validate_group_structure_v3(normalized) + if isinstance(normalized, Mapping): + document = cast("Mapping[str, object]", normalized) + problems = problems + run_rules(ZARR_V3_GROUP_RULES, document) + return tuple(problems) + + +def is_group_metadata_v3(value: object) -> bool: + """Whether `value` is a structurally and compositionally valid v3 group doc. + + Deliberately not a `TypeIs` guard — see the module docstring. + """ + return len(validate_group_metadata_v3(value)) == 0 + + +def parse_group_metadata_v3(value: object) -> ZarrV3GroupMetadataJSON: + """Return `value` as a valid `ZarrV3GroupMetadataJSON`, or raise.""" + normalized = arrays_to_tuples(value) + problems = validate_group_metadata_v3(normalized) + if len(problems) != 0: + raise MetadataValidationError(problems) + return cast("ZarrV3GroupMetadataJSON", normalized) + + +def validate_group_metadata_v2(value: object) -> tuple[ValidationProblem, ...]: + """Every reason `value` is not a valid v2 group document (merged form). + + v2 group documents carry no composition constraints today, so this is + the structural judgment, offered here for a uniform read-side API. + """ + return _validate_group_structure_v2(arrays_to_tuples(value)) + + +def is_group_metadata_v2(value: object) -> bool: + """Whether `value` is a valid v2 group document (merged form).""" + return len(validate_group_metadata_v2(value)) == 0 + + +def parse_group_metadata_v2(value: object) -> ZarrV2GroupMetadataJSON: + """Return `value` as a valid `ZarrV2GroupMetadataJSON`, or raise.""" + normalized = arrays_to_tuples(value) + problems = validate_group_metadata_v2(normalized) + if len(problems) != 0: + raise MetadataValidationError(problems) + return cast("ZarrV2GroupMetadataJSON", normalized) + + +__all__ = [ + "is_array_metadata_v2", + "is_array_metadata_v3", + "is_group_metadata_v2", + "is_group_metadata_v3", + "parse_array_metadata_v2", + "parse_array_metadata_v3", + "parse_group_metadata_v2", + "parse_group_metadata_v3", + "validate_array_metadata_v2", + "validate_array_metadata_v3", + "validate_group_metadata_v2", + "validate_group_metadata_v3", +] diff --git a/packages/zarr-metadata/src/zarr_metadata/rules/_engine.py b/packages/zarr-metadata/src/zarr_metadata/rules/_engine.py new file mode 100644 index 0000000000..bb101bd301 --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/rules/_engine.py @@ -0,0 +1,115 @@ +"""Rules gated by the document fields they read. + +A `Rule` runs when every key in `requires` is present. The same rule set +therefore supports complete documents and partial builders without +imposing field order. + +Prior art +--------- +The gate is conventional, which is the point. Ecto's +`Ecto.Changeset.validate_change/3` invokes a validator "only if a change +for the given field exists", so one changeset serves both full inserts +and partial updates. Clojure spec's two-phase `s/keys` separates +required-key presence from key/value conformance precisely because "we +routinely deal with optional and partial data". Valibot's `partialCheck` +takes the paths a cross-field rule reads and runs it "whenever the +selected part of the data is valid". Presence-conditional rules-as-data +are JSON Schema's `dependentSchemas` / `dependentRequired` applicators. + +- https://hexdocs.pm/ecto/Ecto.Changeset.html +- https://clojure.org/about/spec +- https://valibot.dev/api/partialCheck/ +- https://www.learnjsonschema.com/2020-12/applicator/dependentschemas/ + +Two consequences follow from gating rather than ordering. + +**Order-free by construction.** No topological sort, so mutually +dependent rules are expressible — unlike Yup, whose equivalent `deps` +orders rules and therefore rejects cycles outright. + +**Absence is deliberately inexpressible.** A rule cannot ask whether a +field is missing: that is negation-as-failure, sound only under a +closed-world assumption, and a partially built document is an open world +where the key may still arrive. Required-key checks therefore stay in +structural validation — the same stratification Ecto +(`validate_required`), spec (`:req`), and JSON Schema (`required`) apply. + +Rules may receive structurally invalid values. A rule that cannot safely +interpret its inputs leaves the problem to structural validation. +""" + +from __future__ import annotations + +from collections.abc import Callable, Iterator, Mapping, Sequence +from collections.abc import Set as AbstractSet +from dataclasses import dataclass +from typing import cast + +from zarr_metadata.model._validation import ValidationProblem + +RuleCheck = Callable[[Mapping[str, object]], tuple[ValidationProblem, ...]] +"""A rule's check: the whole document in, every problem it finds out.""" + + +@dataclass(frozen=True, slots=True) +class Rule: + """One composition check over a (possibly partial) metadata document. + + `requires` are the document keys the check reads; the rule fires only + when all of them are present. `check` receives the whole document (so + coupled fields are examined together) and returns every problem it + finds, empty when the rule passes. + """ + + requires: frozenset[str] + check: RuleCheck + + +def applicable(rules: Sequence[Rule], present: AbstractSet[str]) -> Iterator[Rule]: + """The subset of `rules` whose required keys are all present.""" + return (rule for rule in rules if rule.requires <= present) + + +def run_rules( + rules: Sequence[Rule], document: Mapping[str, object] +) -> tuple[ValidationProblem, ...]: + """Run every applicable rule over `document`, collecting all problems.""" + problems: list[ValidationProblem] = [] + for rule in applicable(rules, document.keys()): + problems.extend(rule.check(document)) + return tuple(problems) + + +def as_string_mapping(value: object) -> Mapping[str, object] | None: + """`value` as a string-keyed mapping, or None if it is not one.""" + if not isinstance(value, Mapping): + return None + mapping = cast("Mapping[object, object]", value) + if any(not isinstance(key, str) for key in mapping): + return None + return cast("Mapping[str, object]", mapping) + + +def as_sequence(value: object) -> Sequence[object] | None: + """`value` as a JSON-array-shaped sequence, or None if it is not one.""" + if isinstance(value, (list, tuple)): + return cast("Sequence[object]", value) + return None + + +def prefixed( + loc: tuple[str | int, ...], problems: Sequence[ValidationProblem] +) -> tuple[ValidationProblem, ...]: + """Re-base every problem's `loc` under `loc` (for nested documents).""" + return tuple( + ValidationProblem((*loc, *problem.loc), problem.message, problem.kind) + for problem in problems + ) + + +__all__ = [ + "Rule", + "RuleCheck", + "applicable", + "run_rules", +] diff --git a/packages/zarr-metadata/src/zarr_metadata/rules/_entities/__init__.py b/packages/zarr-metadata/src/zarr_metadata/rules/_entities/__init__.py new file mode 100644 index 0000000000..a1662ccc1f --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/rules/_entities/__init__.py @@ -0,0 +1,27 @@ +"""Per-entity composition rules, discovered automatically. + +Every module here owns the rules for one codec or chunk grid and +registers them with `zarr_metadata.rules._registry` at import time. +Adding a new entity means adding a module here and nothing else: this +package imports every sibling module on import, so there is no +registration list to update and no document-rule module to edit. + +That auto-discovery is the deliberate answer to two failure modes. A +hand-written registry lets a rule be defined and never registered, and a +hand-written import list lets a whole module be defined and never +imported; both produce rules that silently never run. `tests/rules/ +test_registry.py` closes the remaining gap by asserting that every codec +and chunk grid the package models is either registered here or listed as +deliberately rule-free. +""" + +from __future__ import annotations + +import importlib +import pkgutil + +for _module in pkgutil.iter_modules(__path__): + if not _module.name.startswith("_"): + importlib.import_module(f"{__name__}.{_module.name}") + +__all__: list[str] = [] diff --git a/packages/zarr-metadata/src/zarr_metadata/rules/_entities/bytes_codec.py b/packages/zarr-metadata/src/zarr_metadata/rules/_entities/bytes_codec.py new file mode 100644 index 0000000000..048ce33fc7 --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/rules/_entities/bytes_codec.py @@ -0,0 +1,116 @@ +"""Composition rules for the core ``bytes`` codec.""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Literal, cast + +from zarr_metadata.model._validation import ValidationProblem +from zarr_metadata.rules._engine import as_string_mapping +from zarr_metadata.rules._registry import entity_rule +from zarr_metadata.v3._extension_points import CODECS, DATA_TYPE +from zarr_metadata.v3._shape import blocking_problems, validate_known_entity_metadata +from zarr_metadata.v3.codec.bytes import BYTES_CODEC_NAME +from zarr_metadata.v3.data_type.raw import RAW_BYTES_NAME_PATTERN + +if TYPE_CHECKING: + from collections.abc import Mapping + + from zarr_metadata.rules._spec import ArraySpec + +_ARRAY_V3 = "zarr_v3_array" + +_SINGLE_BYTE = frozenset({"bool", "int8", "uint8"}) +_MULTI_BYTE = frozenset( + { + "int16", + "int32", + "int64", + "uint16", + "uint32", + "uint64", + "float16", + "float32", + "float64", + "complex64", + "complex128", + "numpy.datetime64", + "numpy.timedelta64", + } +) +_VARIABLE_LENGTH = frozenset({"bytes", "string"}) +_StorageClass = Literal["single_byte", "multi_byte", "variable_length"] + + +def _data_type_name(data_type: object) -> str | None: + if isinstance(data_type, str): + return data_type + mapping = as_string_mapping(data_type) + if mapping is None: + return None + name = mapping.get("name") + return name if isinstance(name, str) else None + + +def _storage_class(data_type: object) -> _StorageClass | None: + """Classify known data types by their raw byte representation.""" + name = _data_type_name(data_type) + if name in _SINGLE_BYTE or (name is not None and RAW_BYTES_NAME_PATTERN.fullmatch(name)): + return "single_byte" + if name in _MULTI_BYTE: + return "multi_byte" + if name in _VARIABLE_LENGTH: + return "variable_length" + if name != "struct": + return None + + envelope = as_string_mapping(data_type) + configuration = ( + as_string_mapping(envelope.get("configuration")) if envelope is not None else None + ) + fields = configuration.get("fields") if configuration is not None else None + if not isinstance(fields, tuple): + return None + classes: list[_StorageClass] = [] + for field in cast("tuple[object, ...]", fields): + field_mapping = as_string_mapping(field) + if field_mapping is None or "data_type" not in field_mapping: + return None + field_class = _storage_class(field_mapping["data_type"]) + if field_class is None: + return None + classes.append(field_class) + if "variable_length" in classes: + return "variable_length" + if "multi_byte" in classes: + return "multi_byte" + return "single_byte" + + +@entity_rule(_ARRAY_V3, CODECS, BYTES_CODEC_NAME) +def data_type_has_a_raw_byte_representation( + configuration: Mapping[str, object], document: Mapping[str, object], incoming: ArraySpec +) -> tuple[ValidationProblem, ...]: + if incoming.data_type is None: + return () + shape_verdict = validate_known_entity_metadata(DATA_TYPE, incoming.data_type) + if shape_verdict is not None and len(blocking_problems(shape_verdict)) != 0: + return () + storage_class = _storage_class(incoming.data_type) + if storage_class == "variable_length": + name = _data_type_name(incoming.data_type) + return ( + ValidationProblem( + (), + f"bytes codec is not compatible with variable-length data_type {name!r}", + "invalid_value", + ), + ) + if storage_class == "multi_byte" and "endian" not in configuration: + return ( + ValidationProblem( + ("endian",), + "endian is required for a data type containing multi-byte values", + "missing_key", + ), + ) + return () diff --git a/packages/zarr-metadata/src/zarr_metadata/rules/_entities/cast_value.py b/packages/zarr-metadata/src/zarr_metadata/rules/_entities/cast_value.py new file mode 100644 index 0000000000..99f2964eb4 --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/rules/_entities/cast_value.py @@ -0,0 +1,32 @@ +"""Spec transition for the `cast_value` codec. + +`cast_value` carries no composition rules of its own today, but it +changes the data type everything downstream receives: a later rule that +reads the type (the `bytes` codec's endianness requirement, for example) +must judge against the configured target. + +The codec also casts the fill value, and the spec makes a failed +round-trip a MUST error. Deciding that means implementing the cast +(rounding modes, out-of-range clamp and wrap, scalar maps), which is +numeric semantics rather than JSON judgment; it belongs to whatever +implements the codec and is not modelled here. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, cast + +from zarr_metadata.rules._spec import ArraySpec, spec_transition +from zarr_metadata.v3.codec.cast_value import CAST_VALUE_CODEC_NAME + +if TYPE_CHECKING: + from collections.abc import Mapping + + from zarr_metadata.v3._common import ZarrV3MetadataFieldJSON + + +@spec_transition(CAST_VALUE_CODEC_NAME) +def cast_data_type(configuration: Mapping[str, object], incoming: ArraySpec) -> ArraySpec: + """The outgoing type is the configured target.""" + target = cast("ZarrV3MetadataFieldJSON", configuration["data_type"]) + return incoming.with_data_type(target) diff --git a/packages/zarr-metadata/src/zarr_metadata/rules/_entities/gzip.py b/packages/zarr-metadata/src/zarr_metadata/rules/_entities/gzip.py new file mode 100644 index 0000000000..542574d1ad --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/rules/_entities/gzip.py @@ -0,0 +1,31 @@ +"""Composition rules for the core ``gzip`` codec.""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, cast + +from zarr_metadata.model._validation import ValidationProblem +from zarr_metadata.rules._registry import entity_rule +from zarr_metadata.v3._extension_points import CODECS +from zarr_metadata.v3.codec.gzip import GZIP_CODEC_NAME + +if TYPE_CHECKING: + from collections.abc import Mapping + + from zarr_metadata.rules._spec import ArraySpec + +_ARRAY_V3 = "zarr_v3_array" + + +@entity_rule(_ARRAY_V3, CODECS, GZIP_CODEC_NAME) +def level_is_in_range( + configuration: Mapping[str, object], document: Mapping[str, object], incoming: ArraySpec +) -> tuple[ValidationProblem, ...]: + level = cast("int", configuration["level"]) + if 0 <= level <= 9: + return () + return ( + ValidationProblem( + ("level",), f"expected an integer in [0, 9], got {level}", "invalid_value" + ), + ) diff --git a/packages/zarr-metadata/src/zarr_metadata/rules/_entities/numpy_time.py b/packages/zarr-metadata/src/zarr_metadata/rules/_entities/numpy_time.py new file mode 100644 index 0000000000..c5baed7d4e --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/rules/_entities/numpy_time.py @@ -0,0 +1,38 @@ +"""Composition rules shared by NumPy datetime and timedelta data types.""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, cast + +from zarr_metadata.model._validation import ValidationProblem +from zarr_metadata.rules._registry import entity_rule +from zarr_metadata.v3._extension_points import DATA_TYPE +from zarr_metadata.v3.data_type.numpy_datetime64 import NUMPY_DATETIME64_DATA_TYPE_NAME +from zarr_metadata.v3.data_type.numpy_timedelta64 import NUMPY_TIMEDELTA64_DATA_TYPE_NAME + +if TYPE_CHECKING: + from collections.abc import Mapping + + from zarr_metadata.rules._spec import ArraySpec + +_ARRAY_V3 = "zarr_v3_array" +_MAX_SCALE_FACTOR = 2**31 - 1 + + +def _scale_factor_is_in_range( + configuration: Mapping[str, object], document: Mapping[str, object], incoming: ArraySpec +) -> tuple[ValidationProblem, ...]: + scale_factor = cast("int", configuration["scale_factor"]) + if 1 <= scale_factor <= _MAX_SCALE_FACTOR: + return () + return ( + ValidationProblem( + ("scale_factor",), + f"expected an integer in [1, {_MAX_SCALE_FACTOR}], got {scale_factor}", + "invalid_value", + ), + ) + + +entity_rule(_ARRAY_V3, DATA_TYPE, NUMPY_DATETIME64_DATA_TYPE_NAME)(_scale_factor_is_in_range) +entity_rule(_ARRAY_V3, DATA_TYPE, NUMPY_TIMEDELTA64_DATA_TYPE_NAME)(_scale_factor_is_in_range) diff --git a/packages/zarr-metadata/src/zarr_metadata/rules/_entities/rectilinear_grid.py b/packages/zarr-metadata/src/zarr_metadata/rules/_entities/rectilinear_grid.py new file mode 100644 index 0000000000..15f170ee97 --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/rules/_entities/rectilinear_grid.py @@ -0,0 +1,125 @@ +"""Composition rules for the `rectilinear` chunk grid.""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, cast + +from zarr_metadata.model._validation import ValidationProblem +from zarr_metadata.rules._registry import entity_rule +from zarr_metadata.v3._extension_points import CHUNK_GRID +from zarr_metadata.v3.chunk_grid.rectilinear import RECTILINEAR_CHUNK_GRID_NAME + +if TYPE_CHECKING: + from collections.abc import Mapping + + from zarr_metadata.rules._spec import ArraySpec, Sequence + +_ARRAY_V3 = "zarr_v3_array" + + +def _is_int(value: object) -> bool: + return isinstance(value, int) and not isinstance(value, bool) + + +def _expanded_extent(spec: Sequence[object]) -> int | None: + """The total extent an explicit dimension spec covers, or None. + + Entries are chunk sizes or `[size, count]` run-length pairs. Answers + None when any entry is non-positive — the values rule owns that + complaint, and a sum over bad entries would be noise. + """ + total = 0 + for item in spec: + if _is_int(item) and cast(int, item) >= 1: + total += cast(int, item) + elif isinstance(item, tuple): + size, count = cast("tuple[object, object]", item) + if not (_is_int(size) and _is_int(count)): + return None + if cast(int, size) < 1 or cast(int, count) < 1: + return None + total += cast(int, size) * cast(int, count) + else: + return None + return total + + +@entity_rule(_ARRAY_V3, CHUNK_GRID, RECTILINEAR_CHUNK_GRID_NAME) +def chunk_extents_are_positive( + configuration: Mapping[str, object], document: Mapping[str, object], incoming: ArraySpec +) -> tuple[ValidationProblem, ...]: + """Every chunk extent, bare or run-length encoded, must be positive.""" + chunk_shapes = cast("tuple[object, ...]", configuration["chunk_shapes"]) + problems: list[ValidationProblem] = [] + for dim, spec in enumerate(chunk_shapes): + loc: tuple[str | int, ...] = ("chunk_shapes", dim) + if _is_int(spec): + if cast(int, spec) < 1: + problems.append( + ValidationProblem( + loc, f"expected a positive chunk extent, got {spec}", "invalid_value" + ) + ) + continue + if not isinstance(spec, tuple): + continue + for position, item in enumerate(cast("tuple[object, ...]", spec)): + if _is_int(item) and cast(int, item) < 1: + problems.append( + ValidationProblem( + (*loc, position), + f"expected a positive chunk extent, got {item}", + "invalid_value", + ) + ) + elif isinstance(item, tuple): + size, count = cast("tuple[int, int]", item) + if size < 1 or count < 1: + problems.append( + ValidationProblem( + (*loc, position), + f"expected a positive [size, count] pair, got {item!r}", + "invalid_value", + ) + ) + return tuple(problems) + + +@entity_rule(_ARRAY_V3, CHUNK_GRID, RECTILINEAR_CHUNK_GRID_NAME, requires=frozenset({"shape"})) +def tiles_the_array( + configuration: Mapping[str, object], document: Mapping[str, object], incoming: ArraySpec +) -> tuple[ValidationProblem, ...]: + """One spec per dimension, and explicit specs must sum to that extent. + + A bare-integer dimension spec is uniform shorthand and imposes no sum + constraint; an explicit list of chunk sizes must tile its dimension + exactly. + """ + shape = document["shape"] + if not isinstance(shape, (list, tuple)): + return () + extents = cast("tuple[object, ...]", shape) + chunk_shapes = cast("tuple[object, ...]", configuration["chunk_shapes"]) + if len(chunk_shapes) != len(extents): + return ( + ValidationProblem( + ("chunk_shapes",), + f"chunk_shapes has {len(chunk_shapes)} entries but shape has " + f"{len(extents)} dimensions", + "invalid_value", + ), + ) + problems: list[ValidationProblem] = [] + for dim, (spec, extent) in enumerate(zip(chunk_shapes, extents, strict=True)): + if not _is_int(extent) or _is_int(spec) or not isinstance(spec, tuple): + continue + total = _expanded_extent(cast("tuple[object, ...]", spec)) + if total is not None and total < cast("int", extent): + problems.append( + ValidationProblem( + ("chunk_shapes", dim), + f"chunk sizes sum to {total} but must cover shape[{dim}] extent {extent}", + "invalid_value", + ) + ) + return tuple(problems) diff --git a/packages/zarr-metadata/src/zarr_metadata/rules/_entities/regular_grid.py b/packages/zarr-metadata/src/zarr_metadata/rules/_entities/regular_grid.py new file mode 100644 index 0000000000..0a19b10133 --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/rules/_entities/regular_grid.py @@ -0,0 +1,60 @@ +"""Composition rules for the `regular` chunk grid.""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, cast + +from zarr_metadata.model._validation import ValidationProblem +from zarr_metadata.rules._registry import entity_rule +from zarr_metadata.v3._extension_points import CHUNK_GRID +from zarr_metadata.v3.chunk_grid.regular import REGULAR_CHUNK_GRID_NAME + +if TYPE_CHECKING: + from collections.abc import Mapping + + from zarr_metadata.rules._spec import ArraySpec + +_ARRAY_V3 = "zarr_v3_array" + + +@entity_rule(_ARRAY_V3, CHUNK_GRID, REGULAR_CHUNK_GRID_NAME) +def chunk_extents_are_positive( + configuration: Mapping[str, object], document: Mapping[str, object], incoming: ArraySpec +) -> tuple[ValidationProblem, ...]: + """Every chunk extent must be at least one element. + + A zero extent makes the chunk index `floor(i / 0)` undefined; a + negative one is meaningless. The shape validator enforces that the + entries are integers, so this rule judges only their values. + """ + chunk_shape = cast("tuple[int, ...]", configuration["chunk_shape"]) + return tuple( + ValidationProblem( + ("chunk_shape", position), + f"expected a positive chunk extent, got {extent}", + "invalid_value", + ) + for position, extent in enumerate(chunk_shape) + if extent < 1 + ) + + +@entity_rule(_ARRAY_V3, CHUNK_GRID, REGULAR_CHUNK_GRID_NAME, requires=frozenset({"shape"})) +def chunks_every_dimension( + configuration: Mapping[str, object], document: Mapping[str, object], incoming: ArraySpec +) -> tuple[ValidationProblem, ...]: + """A regular grid must chunk every array dimension.""" + shape = document["shape"] + if not isinstance(shape, (list, tuple)): + return () + chunk_shape = cast("tuple[int, ...]", configuration["chunk_shape"]) + if len(chunk_shape) == len(cast("tuple[object, ...]", shape)): + return () + return ( + ValidationProblem( + ("chunk_shape",), + f"chunk_shape has {len(chunk_shape)} entries but shape has " + f"{len(cast('tuple[object, ...]', shape))} dimensions", + "invalid_value", + ), + ) diff --git a/packages/zarr-metadata/src/zarr_metadata/rules/_entities/sharding.py b/packages/zarr-metadata/src/zarr_metadata/rules/_entities/sharding.py new file mode 100644 index 0000000000..ccc74c1c01 --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/rules/_entities/sharding.py @@ -0,0 +1,150 @@ +"""Composition rules for the `sharding_indexed` codec. + +Sharding is the one entity whose configuration contains whole pipelines +and its own geometry, so its rules recurse: the inner `codecs` and +`index_codecs` are judged by the same pipeline checks that judge the +document's top-level `codecs`, at every nesting depth. + +Every geometry judgment here is against the *incoming* array spec — the +array as transformed by every codec before this one — never against the +document's chunk grid directly. A `transpose` in front of a shard changes +which extents the shard has to divide, and reading the grid instead gave +wrong verdicts in both directions: it accepted an inner chunk that did +not divide the transposed shape and rejected one that did. + +The inner pipeline receives the inner chunk as its incoming spec (with +the incoming data type carried through), so a transpose or nested shard +inside it is judged against the inner chunk, recursively — each sharding +level encloses the next. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, cast + +from zarr_metadata.model._validation import ValidationProblem +from zarr_metadata.rules._pipeline import pipeline_order_problems, shape_problems +from zarr_metadata.rules._registry import entity_rule, run_chain_rules +from zarr_metadata.rules._spec import NOTHING_KNOWN, ArraySpec +from zarr_metadata.v3._extension_points import CODECS +from zarr_metadata.v3._shape import entity_name +from zarr_metadata.v3.codec.blosc import BLOSC_CODEC_NAME +from zarr_metadata.v3.codec.gzip import GZIP_CODEC_NAME +from zarr_metadata.v3.codec.sharding_indexed import SHARDING_INDEXED_CODEC_NAME +from zarr_metadata.v3.codec.zstd import ZSTD_CODEC_NAME + +if TYPE_CHECKING: + from collections.abc import Mapping + +_ARRAY_V3 = "zarr_v3_array" +_VARIABLE_SIZE_CODECS = frozenset( + {BLOSC_CODEC_NAME, GZIP_CODEC_NAME, SHARDING_INDEXED_CODEC_NAME, ZSTD_CODEC_NAME} +) + + +@entity_rule(_ARRAY_V3, CODECS, SHARDING_INDEXED_CODEC_NAME) +def inner_chunk_extents_are_positive( + configuration: Mapping[str, object], document: Mapping[str, object], incoming: ArraySpec +) -> tuple[ValidationProblem, ...]: + chunk_shape = cast("tuple[int, ...]", configuration["chunk_shape"]) + return tuple( + ValidationProblem( + ("chunk_shape", position), + f"expected a positive chunk extent, got {extent}", + "invalid_value", + ) + for position, extent in enumerate(chunk_shape) + if extent < 1 + ) + + +@entity_rule(_ARRAY_V3, CODECS, SHARDING_INDEXED_CODEC_NAME) +def inner_chunks_tile_the_incoming_array( + configuration: Mapping[str, object], document: Mapping[str, object], incoming: ArraySpec +) -> tuple[ValidationProblem, ...]: + """The inner chunk must rank-match and evenly divide the array it receives. + + Declines when the incoming shape is unknown — an unclassified codec + upstream, or a non-regular grid at the top level — rather than + guessing from the document. + """ + if incoming.shape is None: + return () + outer = incoming.shape + inner = cast("tuple[int, ...]", configuration["chunk_shape"]) + if len(inner) != len(outer): + return ( + ValidationProblem( + ("chunk_shape",), + f"chunk_shape has {len(inner)} entries but the incoming array has " + f"{len(outer)} dimensions", + "invalid_value", + ), + ) + return tuple( + ValidationProblem( + ("chunk_shape", position), + f"inner chunk extent {inner_extent} does not evenly divide the " + f"incoming extent {outer_extent}", + "invalid_value", + ) + for position, (outer_extent, inner_extent) in enumerate(zip(outer, inner, strict=True)) + if inner_extent >= 1 and outer_extent % inner_extent != 0 + ) + + +@entity_rule(_ARRAY_V3, CODECS, SHARDING_INDEXED_CODEC_NAME) +def inner_pipelines_are_pipelines( + configuration: Mapping[str, object], document: Mapping[str, object], incoming: ArraySpec +) -> tuple[ValidationProblem, ...]: + """`codecs` and `index_codecs` obey the pipeline rules, recursively. + + Both get the ordering and shape judgments the top-level pipeline gets, + plus the entity rules of whatever codecs appear inside. The inner + `codecs` chain starts from the inner chunk with the incoming data + type; a nested shard or transpose inside it is therefore judged + against the inner chunk, and its own transitions carry on from there. + The `index_codecs` chain encodes the shard index, a `uint64` array + whose shape this package does not compute. + """ + inner_shape = configuration["chunk_shape"] + if not isinstance(inner_shape, tuple) or not all( + isinstance(v, int) and not isinstance(v, bool) and v >= 1 + for v in cast("tuple[object, ...]", inner_shape) + ): + inner_start = NOTHING_KNOWN + else: + # The inner pipeline encodes the inner chunk: same type as arrived + # here, shape of one inner chunk. + inner_start = incoming.with_shape(cast("tuple[int, ...]", inner_shape)) + problems: list[ValidationProblem] = [] + for key in ("codecs", "index_codecs"): + entries = configuration[key] + if not isinstance(entries, (list, tuple)): + continue + sequence = cast("tuple[object, ...]", entries) + problems.extend(pipeline_order_problems(sequence, (key,))) + problems.extend(shape_problems(sequence, (key,))) + # The index pipeline encodes the shard index, not the array: a + # uint64 array of offsets and lengths, so e.g. the bytes codec + # inside it still needs an endianness. + start = inner_start if key == "codecs" else ArraySpec(None, "uint64") + problems.extend(run_chain_rules(CODECS, sequence, document, (key,), start)) + return tuple(problems) + + +@entity_rule(_ARRAY_V3, CODECS, SHARDING_INDEXED_CODEC_NAME) +def index_codecs_have_fixed_encoded_size( + configuration: Mapping[str, object], document: Mapping[str, object], incoming: ArraySpec +) -> tuple[ValidationProblem, ...]: + """The shard index must have an encoded size derivable from metadata.""" + entries = cast("tuple[object, ...]", configuration["index_codecs"]) + return tuple( + ValidationProblem( + ("index_codecs", index), + f"{name!r} produces variable-size output; index_codecs must be fixed-size", + "invalid_value", + ) + for index, entry in enumerate(entries) + if (name := entity_name(entry)) in _VARIABLE_SIZE_CODECS + ) diff --git a/packages/zarr-metadata/src/zarr_metadata/rules/_entities/struct_dtype.py b/packages/zarr-metadata/src/zarr_metadata/rules/_entities/struct_dtype.py new file mode 100644 index 0000000000..f0739735d1 --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/rules/_entities/struct_dtype.py @@ -0,0 +1,164 @@ +"""Composition rules for the `struct` data type. + +`StructField`'s own docstring promises field names are unique within a +struct and non-empty. Neither is expressible in a TypedDict, so both are +composition judgments and live here. +""" + +from __future__ import annotations + +from collections.abc import Mapping +from typing import TYPE_CHECKING, cast + +from zarr_metadata.model._validation import ValidationProblem +from zarr_metadata.rules._engine import as_string_mapping +from zarr_metadata.rules._registry import entity_rule +from zarr_metadata.v3._extension_points import DATA_TYPE +from zarr_metadata.v3.data_type.raw import RAW_BYTES_NAME_PATTERN +from zarr_metadata.v3.data_type.struct import STRUCT_DATA_TYPE_NAME + +if TYPE_CHECKING: + from zarr_metadata.rules._spec import ArraySpec + +_ARRAY_V3 = "zarr_v3_array" + +_FIXED_SIZE_NAMES = frozenset( + { + "bool", + "int8", + "int16", + "int32", + "int64", + "uint8", + "uint16", + "uint32", + "uint64", + "float16", + "float32", + "float64", + "complex64", + "complex128", + "numpy.datetime64", + "numpy.timedelta64", + } +) +_VARIABLE_SIZE_NAMES = frozenset({"bytes", "string"}) + + +def _field_names(configuration: Mapping[str, object]) -> tuple[tuple[int, str], ...]: + """`(index, name)` for each field with a string name, else nothing. + + Anything the shape validator would reject is skipped: it owns that + complaint, and judging names inside a malformed field list is noise. + """ + fields = configuration.get("fields") + if not isinstance(fields, tuple): + return () + named: list[tuple[int, str]] = [] + for index, field in enumerate(cast("tuple[object, ...]", fields)): + if not isinstance(field, Mapping): + continue + name = cast("Mapping[object, object]", field).get("name") + if isinstance(name, str): + named.append((index, name)) + return tuple(named) + + +def _known_fixed_size(data_type: object) -> bool | None: + """Whether a known data type is fixed-size; None means unknown.""" + if isinstance(data_type, str): + name = data_type + envelope = None + else: + envelope = as_string_mapping(data_type) + raw_name = envelope.get("name") if envelope is not None else None + name = raw_name if isinstance(raw_name, str) else None + if name in _FIXED_SIZE_NAMES or ( + isinstance(name, str) and RAW_BYTES_NAME_PATTERN.fullmatch(name) + ): + return True + if name in _VARIABLE_SIZE_NAMES: + return False + if name != STRUCT_DATA_TYPE_NAME or envelope is None: + return None + nested_configuration = as_string_mapping(envelope.get("configuration")) + fields = nested_configuration.get("fields") if nested_configuration is not None else None + if not isinstance(fields, tuple): + return None + results: list[bool] = [] + for field in cast("tuple[object, ...]", fields): + field_mapping = as_string_mapping(field) + if field_mapping is None or "data_type" not in field_mapping: + return None + result = _known_fixed_size(field_mapping["data_type"]) + if result is None: + return None + results.append(result) + return all(results) + + +@entity_rule(_ARRAY_V3, DATA_TYPE, STRUCT_DATA_TYPE_NAME) +def fields_are_non_empty( + configuration: Mapping[str, object], document: Mapping[str, object], incoming: ArraySpec +) -> tuple[ValidationProblem, ...]: + fields = cast("tuple[object, ...]", configuration["fields"]) + if len(fields) != 0: + return () + return (ValidationProblem(("fields",), "expected at least one struct field", "invalid_value"),) + + +@entity_rule(_ARRAY_V3, DATA_TYPE, STRUCT_DATA_TYPE_NAME) +def field_data_types_are_fixed_size( + configuration: Mapping[str, object], document: Mapping[str, object], incoming: ArraySpec +) -> tuple[ValidationProblem, ...]: + fields = cast("tuple[object, ...]", configuration["fields"]) + problems: list[ValidationProblem] = [] + for index, field in enumerate(fields): + field_mapping = as_string_mapping(field) + if field_mapping is None or "data_type" not in field_mapping: + continue + if _known_fixed_size(field_mapping["data_type"]) is False: + problems.append( + ValidationProblem( + ("fields", index, "data_type"), + "struct fields must use fixed-size data types", + "invalid_value", + ) + ) + return tuple(problems) + + +@entity_rule(_ARRAY_V3, DATA_TYPE, STRUCT_DATA_TYPE_NAME) +def field_names_are_non_empty( + configuration: Mapping[str, object], document: Mapping[str, object], incoming: ArraySpec +) -> tuple[ValidationProblem, ...]: + """A struct field must be addressable, so its name cannot be empty.""" + return tuple( + ValidationProblem( + ("fields", index, "name"), "expected a non-empty field name", "invalid_value" + ) + for index, name in _field_names(configuration) + if name == "" + ) + + +@entity_rule(_ARRAY_V3, DATA_TYPE, STRUCT_DATA_TYPE_NAME) +def field_names_are_unique( + configuration: Mapping[str, object], document: Mapping[str, object], incoming: ArraySpec +) -> tuple[ValidationProblem, ...]: + """Duplicate field names make a fill value's per-field mapping ambiguous.""" + seen: dict[str, int] = {} + problems: list[ValidationProblem] = [] + for index, name in _field_names(configuration): + first = seen.get(name) + if first is None: + seen[name] = index + continue + problems.append( + ValidationProblem( + ("fields", index, "name"), + f"duplicate field name {name!r}, already used by field {first}", + "invalid_value", + ) + ) + return tuple(problems) diff --git a/packages/zarr-metadata/src/zarr_metadata/rules/_entities/transpose.py b/packages/zarr-metadata/src/zarr_metadata/rules/_entities/transpose.py new file mode 100644 index 0000000000..61d98f0b4b --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/rules/_entities/transpose.py @@ -0,0 +1,78 @@ +"""Composition rules and spec transition for the `transpose` codec.""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, cast + +from zarr_metadata.model._validation import ValidationProblem +from zarr_metadata.rules._registry import entity_rule +from zarr_metadata.rules._spec import ArraySpec, spec_transition +from zarr_metadata.v3._extension_points import CODECS +from zarr_metadata.v3.codec.transpose import TRANSPOSE_CODEC_NAME + +if TYPE_CHECKING: + from collections.abc import Mapping + +_ARRAY_V3 = "zarr_v3_array" + + +@spec_transition(TRANSPOSE_CODEC_NAME) +def permute_shape(configuration: Mapping[str, object], incoming: ArraySpec) -> ArraySpec: + """The outgoing shape is the incoming shape permuted by `order`. + + Declines (shape None) when the order is not a permutation of the + incoming rank: the rules below report that, and any shape derived + from a bad order would be a guess. + """ + order = cast("tuple[int, ...]", configuration["order"]) + shape = incoming.shape + if shape is None or sorted(order) != list(range(len(shape))): + return incoming.with_shape(None) + return incoming.with_shape(tuple(shape[axis] for axis in order)) + + +@entity_rule(_ARRAY_V3, CODECS, TRANSPOSE_CODEC_NAME) +def order_is_a_permutation( + configuration: Mapping[str, object], document: Mapping[str, object], incoming: ArraySpec +) -> tuple[ValidationProblem, ...]: + """`order` must be a permutation of its own indices. + + Checked without reference to the incoming shape, so it holds even + when propagation has stopped upstream. + """ + order = cast("tuple[int, ...]", configuration["order"]) + if sorted(order) == list(range(len(order))): + return () + return ( + ValidationProblem( + ("order",), + f"expected a permutation of 0..{len(order) - 1}, got {order!r}", + "invalid_value", + ), + ) + + +@entity_rule(_ARRAY_V3, CODECS, TRANSPOSE_CODEC_NAME) +def order_matches_incoming_rank( + configuration: Mapping[str, object], document: Mapping[str, object], incoming: ArraySpec +) -> tuple[ValidationProblem, ...]: + """A transpose permutes the array it receives, so ranks must agree. + + Judged against the *incoming* spec, not the document's `shape`: inside + a shard the incoming array is the inner chunk, and after another + transpose it is that transpose's output. Declines when the incoming + shape is unknown. + """ + if incoming.shape is None: + return () + order = cast("tuple[int, ...]", configuration["order"]) + if len(order) == len(incoming.shape): + return () + return ( + ValidationProblem( + ("order",), + f"order has {len(order)} entries but the incoming array has " + f"{len(incoming.shape)} dimensions", + "invalid_value", + ), + ) diff --git a/packages/zarr-metadata/src/zarr_metadata/rules/_pipeline.py b/packages/zarr-metadata/src/zarr_metadata/rules/_pipeline.py new file mode 100644 index 0000000000..34a05deb50 --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/rules/_pipeline.py @@ -0,0 +1,119 @@ +"""Codec-pipeline judgments, shared by the array rules and by sharding. + +A sharding codec's `codecs` and `index_codecs` are pipelines exactly like +the document's top-level `codecs`, so the ordering and shape checks live +here rather than in either caller: sharding recurses into them at every +nesting depth, and the top-level array rules apply them at depth zero. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Final + +from zarr_metadata.model._validation import ValidationProblem +from zarr_metadata.rules._engine import as_string_mapping, prefixed +from zarr_metadata.v3._shape import entity_name, validate_known_codec_metadata +from zarr_metadata.v3.codec.kind import codec_kind_of_name + +if TYPE_CHECKING: + from collections.abc import Sequence + + from zarr_metadata.v3.codec.kind import CodecKind + +_KIND_RANK: Final = {"array_array": 0, "array_bytes": 1, "bytes_bytes": 2} + + +def codec_kind(codec: object) -> CodecKind | None: + """The pipeline kind of `codec`, classified by name alone. + + Spelling-insensitive on purpose: a known codec in an invalid spelling + still ranks as its kind, so two spellings of the same pipeline always + get the same ordering verdict and a misspelled known codec is never + mistaken for an unknown extension (which would suppress the + exactly-one-`array->bytes` count). + """ + name = entity_name(codec) + if name is None: + return None + return codec_kind_of_name(name) + + +def _codec_label(codec: object) -> str: + if isinstance(codec, str): + return repr(codec) + mapping = as_string_mapping(codec) + if mapping is not None: + return repr(mapping.get("name")) + return repr(codec) + + +def pipeline_order_problems( + entries: Sequence[object], loc: tuple[str | int, ...] +) -> tuple[ValidationProblem, ...]: + """The spec pipeline shape: `array->array`* `array->bytes` `bytes->bytes`*. + + Codecs of genuinely unknown name are skipped: they impose no ordering + constraint, and their presence makes the exactly-one-`array->bytes` + count inconclusive (an unknown codec might be the pipeline's + `array->bytes` stage), so that check only fires when every codec is + classified. + """ + + problems: list[ValidationProblem] = [] + kinds = [codec_kind(codec) for codec in entries] + max_rank_seen = -1 + array_bytes_count = 0 + for index, (codec, kind) in enumerate(zip(entries, kinds, strict=True)): + if kind is None: + continue + rank = _KIND_RANK[kind] + if rank < max_rank_seen: + problems.append( + ValidationProblem( + (*loc, index), + f"{kind.replace('_', '->')} codec {_codec_label(codec)} may not " + "follow a later-stage codec in the pipeline", + "invalid_value", + ) + ) + max_rank_seen = max(max_rank_seen, rank) + if kind == "array_bytes": + array_bytes_count += 1 + if array_bytes_count > 1: + problems.append( + ValidationProblem( + (*loc, index), + f"extra array->bytes codec {_codec_label(codec)}: a pipeline " + "has exactly one", + "invalid_value", + ) + ) + if array_bytes_count == 0 and all(kind is not None for kind in kinds): + problems.append( + ValidationProblem(loc, "codec pipeline has no array->bytes codec", "invalid_value") + ) + return tuple(problems) + + +def shape_problems( + entries: Sequence[object], loc: tuple[str | int, ...] +) -> tuple[ValidationProblem, ...]: + """Shape problems for every known-name codec entry in `entries`. + + Unknown names pass untouched (extension openness); entries without an + interpretable name decline in favor of the structural validator. + """ + problems: list[ValidationProblem] = [] + for index, codec in enumerate(entries): + found = validate_known_codec_metadata(codec) + # None is "not a known codec" (unjudged); () is "known and valid". + if found is not None: + problems.extend(prefixed((*loc, index), found)) + return tuple(problems) + + +__all__ = [ + "codec_kind", + "pipeline_order_problems", + "shape_problems", +] diff --git a/packages/zarr-metadata/src/zarr_metadata/rules/_registry.py b/packages/zarr-metadata/src/zarr_metadata/rules/_registry.py new file mode 100644 index 0000000000..042d3542bb --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/rules/_registry.py @@ -0,0 +1,319 @@ +"""Register rules by document type and extension entity. + +`@document_rule` and `@entity_rule` register checks where they are +defined, so a rule cannot be written without joining the set it belongs +to. Both reject dependencies absent from the document type. Entity rules +are keyed by `(field, canonical_name)` and require a corresponding shape +validator. +""" + +from __future__ import annotations + +from collections import defaultdict +from collections.abc import Callable, Mapping, Sequence +from dataclasses import dataclass +from typing import TYPE_CHECKING, Final, cast + +from zarr_metadata.rules._engine import Rule, as_string_mapping, prefixed +from zarr_metadata.rules._spec import NOTHING_KNOWN, ArraySpec, initial_spec, propagate +from zarr_metadata.v3._extension_points import CHUNK_GRID, ExtensionPointField, canonical_name +from zarr_metadata.v3._shape import ( + blocking_problems, + entity_name, + modelled_entities, + validate_known_entity_metadata, +) + +if TYPE_CHECKING: + from zarr_metadata.model._validation import ValidationProblem + +EntityCheck = Callable[ + [Mapping[str, object], Mapping[str, object], "ArraySpec"], + "tuple[ValidationProblem, ...]", +] +"""An entity rule's check: `(configuration, document, incoming)` in, problems out. + +`incoming` is the `ArraySpec` the entity receives — for a codec, the +array as transformed by every codec before it in the chain. Fields this +package cannot determine are `None`; a caller with no chain context +passes `NOTHING_KNOWN`. Rules that need a field test it `is None` and +decline; rules that do not simply ignore the spec. + +Problems carry locations relative to the entity's `configuration`; the +dispatcher re-bases them onto the entity's position in the document. +""" + + +@dataclass(frozen=True, slots=True) +class EntityRule: + """One composition check for a named extension entity. + + Identified by `(field, entity)`, never by name alone: names are + unique only within an extension point, and `bytes` is both a core + codec and a registered extension data type. Keying by name would + make a rule written for one fire on the other. + + `requires` are *document* keys the check reads beyond the entity + itself (e.g. `shape`), gating the rule exactly as `Rule.requires` + does. + """ + + field: str + entity: str + requires: frozenset[str] + check: EntityCheck + + +_DOCUMENT_RULES: Final[dict[str, list[Rule]]] = defaultdict(list) +_ENTITY_RULES: Final[dict[tuple[str, str], list[EntityRule]]] = defaultdict(list) +_DOCUMENT_KEYS: Final[dict[str, frozenset[str]]] = {} +_DISPATCHED_FIELDS: Final[set[str]] = set() + + +def register_document_type( + document_type: str, + standard_keys: frozenset[str], + extension_keys: frozenset[str] = frozenset(), +) -> None: + """Declare a document type's known keys, so `requires` can be checked. + + `extension_keys` names keys that are not part of the document's + TypedDict but that this package nonetheless recognizes — the v3 + `consolidated_metadata` convention is the only one today. Requiring + them to be declared here rather than exempting unknown keys wholesale + keeps the typo check meaningful. + """ + _DOCUMENT_KEYS[document_type] = standard_keys | extension_keys + + +def _validate_requires(document_type: str, requires: frozenset[str], what: str) -> None: + known = _DOCUMENT_KEYS.get(document_type) + if known is None: + msg = f"unknown document type {document_type!r} registering {what}" + raise LookupError(msg) + unknown = requires - known + if len(unknown) != 0: + msg = ( + f"{what} requires {sorted(unknown)}, which {document_type} documents " + f"do not have; such a rule could never fire" + ) + raise ValueError(msg) + + +def document_rule( + document_type: str, requires: frozenset[str] +) -> Callable[[Callable[[Mapping[str, object]], tuple[ValidationProblem, ...]]], Rule]: + """Register a whole-document rule, returning the `Rule` it becomes. + + The decorated function is replaced by its `Rule`, so a rule cannot be + defined without being registered, and referencing one by name yields + the registered object rather than a copy. + """ + + def decorate( + check: Callable[[Mapping[str, object]], tuple[ValidationProblem, ...]], + ) -> Rule: + _validate_requires(document_type, requires, f"rule {check.__name__!r}") + rule = Rule(requires=requires, check=check) + _DOCUMENT_RULES[document_type].append(rule) + return rule + + return decorate + + +def entity_rule( + document_type: str, + field: ExtensionPointField, + entity: str, + requires: frozenset[str] = frozenset(), +) -> Callable[[EntityCheck], EntityRule]: + """Register a rule about one named entity within `document_type`. + + The entity must already be shape-modelled in `zarr_metadata.v3._shape`: + entity rules read configuration members by name, so they only run once + the shape validator vouches those members exist and are typed. A rule + registered for an unmodelled name would silently never fire, so that + is refused here rather than discovered as a missing check later. + """ + + def decorate(check: EntityCheck) -> EntityRule: + _validate_requires(document_type, requires, f"entity rule {check.__name__!r}") + canonical_entity = canonical_name(field, entity) + if (field, canonical_entity) not in modelled_entities(): + msg = ( + f"entity rule {check.__name__!r} targets {entity!r}, which has no shape " + f"validator in zarr_metadata.v3._shape; such a rule could never fire" + ) + raise ValueError(msg) + rule = EntityRule(field=field, entity=entity, requires=requires, check=check) + _ENTITY_RULES[field, canonical_entity].append(rule) + return rule + + return decorate + + +def document_rules(document_type: str) -> tuple[Rule, ...]: + """Every rule registered for `document_type`, in definition order.""" + return tuple(_DOCUMENT_RULES[document_type]) + + +def dispatched_fields() -> frozenset[str]: + """Extension points that have a dispatcher, so their rules can run. + + An entity rule registered at a field with no dispatcher is accepted and + then never fires — the silent-pass failure this module exists to + prevent. Checking coverage at registration would depend on import + order, so `tests/rules/test_registry.py` asserts it instead. + """ + return frozenset(_DISPATCHED_FIELDS) + + +def registered_entities() -> frozenset[tuple[str, str]]: + """Every `(field, canonical name)` that has at least one registered rule.""" + return frozenset(_ENTITY_RULES) + + +def run_entity_rules( + field: ExtensionPointField, + value: object, + document: Mapping[str, object], + loc: tuple[str | int, ...], + incoming: ArraySpec = NOTHING_KNOWN, +) -> tuple[ValidationProblem, ...]: + """Run the rules registered for whatever entity `value` names. + + Declines silently when `value` names nothing known, when its shape is + broken in a way that makes its configuration uninterpretable (the + shape rule owns that complaint), or when a rule's required document + keys are absent. An `unknown_key` never declines — see + `zarr_metadata.v3._shape.blocking_problems`. + """ + name = entity_name(value) + if name is None: + return () + rules = _ENTITY_RULES.get((field, canonical_name(field, name))) + if rules is None or len(rules) == 0: + return () + # Entity rules read configuration members by name, so they may only run + # once the shape validator vouches those members exist and are typed. + configuration = entity_configuration(field, value) + if configuration is None: + return () + problems: list[ValidationProblem] = [] + for rule in rules: + if not rule.requires <= document.keys(): + continue + problems.extend( + prefixed((*loc, "configuration"), rule.check(configuration, document, incoming)) + ) + return tuple(problems) + + +def entity_configuration(field: ExtensionPointField, value: object) -> Mapping[str, object] | None: + """`value`'s configuration if its modelled fields are usable, else None. + + Shared by the dispatchers and by rules that reach across entities + (sharding's nested pipelines). `unknown_key` problems do not make an + entity unusable; anything else does. + """ + verdict = validate_known_entity_metadata(field, value) + if verdict is None or len(blocking_problems(verdict)) != 0: + return None + mapping = as_string_mapping(value) + if mapping is None: + # Bare-string metadata is the canonical spelling for entities whose + # configuration is optional. Rules still need a real mapping to run + # against, especially when they judge a missing optional member. + return {} if isinstance(value, str) else None + if "configuration" not in mapping: + return {} + return as_string_mapping(mapping["configuration"]) + + +def dispatch_field( + field: ExtensionPointField, +) -> Callable[[Mapping[str, object]], tuple[ValidationProblem, ...]]: + """A check that runs entity rules for the entity in `document[field]`.""" + + def check(document: Mapping[str, object]) -> tuple[ValidationProblem, ...]: + return run_entity_rules(field, document[field], document, (field,)) + + _DISPATCHED_FIELDS.add(field) + check.__name__ = f"_dispatch_{field}_entity_rules" + return check + + +def dispatch_field_sequence( + field: ExtensionPointField, +) -> Callable[[Mapping[str, object]], tuple[ValidationProblem, ...]]: + """A check that runs entity rules for every entity in `document[field]`.""" + + def check(document: Mapping[str, object]) -> tuple[ValidationProblem, ...]: + entries = document[field] + if not isinstance(entries, (list, tuple)): + return () + sequence = cast("tuple[object, ...]", entries) + return run_chain_rules(field, sequence, document, (field,), chain_initial_spec(document)) + + _DISPATCHED_FIELDS.add(field) + check.__name__ = f"_dispatch_{field}_entity_rules" + return check + + +def run_chain_rules( + field: ExtensionPointField, + codecs: Sequence[object], + document: Mapping[str, object], + loc: tuple[str | int, ...], + initial: ArraySpec, +) -> tuple[ValidationProblem, ...]: + """Run entity rules over a codec chain, propagating the array spec. + + Each codec's rules receive the spec that codec actually receives — + the array as transformed by everything before it. Shared by the + top-level `codecs` dispatcher and by sharding, whose inner pipelines + are chains that start from the inner chunk. + """ + problems: list[ValidationProblem] = [] + for index, entry, incoming in propagate( + codecs, initial, lambda codec: entity_configuration(field, codec) + ): + problems.extend(run_entity_rules(field, entry, document, (*loc, index), incoming)) + return tuple(problems) + + +def chain_initial_spec(document: Mapping[str, object]) -> ArraySpec: + """The spec entering a document's top-level codec chain. + + The array a chunk pipeline encodes is one chunk: shape from a regular + grid this package can read (None otherwise), data type from the + document. Non-positive chunk extents yield None for the shape — the + grid's own values rule owns that complaint, and geometry against a + zero extent is noise on top of it. + """ + from zarr_metadata.v3.chunk_grid.regular import REGULAR_CHUNK_GRID_NAME + + grid = document.get("chunk_grid") + chunk_shape: tuple[int, ...] | None = None + if entity_name(grid) == REGULAR_CHUNK_GRID_NAME: + configuration = entity_configuration(CHUNK_GRID, grid) + extents = configuration.get("chunk_shape") if configuration is not None else None + if isinstance(extents, tuple): + values = cast("tuple[object, ...]", extents) + if all(isinstance(v, int) and not isinstance(v, bool) and v >= 1 for v in values): + chunk_shape = cast("tuple[int, ...]", values) + return initial_spec(document, chunk_shape) + + +__all__ = [ + "EntityCheck", + "EntityRule", + "chain_initial_spec", + "dispatched_fields", + "document_rule", + "document_rules", + "entity_rule", + "registered_entities", + "run_chain_rules", + "run_entity_rules", +] diff --git a/packages/zarr-metadata/src/zarr_metadata/rules/_result.py b/packages/zarr-metadata/src/zarr_metadata/rules/_result.py new file mode 100644 index 0000000000..4b099c6947 --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/rules/_result.py @@ -0,0 +1,101 @@ +"""Tagged validation results. + +`check_*` returns `Valid[T] | Invalid`. Testing the literal `valid` +field narrows to either the normalized document or a nonempty problem +tuple. Use `validate_*` to collect problems and `parse_*` to raise on +invalid input. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Generic, Literal, TypeAlias, TypeVar, cast + +from zarr_metadata.model._validation import ValidationProblem, arrays_to_tuples +from zarr_metadata.rules._documents import ( + validate_array_metadata_v2, + validate_array_metadata_v3, + validate_group_metadata_v2, + validate_group_metadata_v3, +) +from zarr_metadata.v2.array import ZarrV2ArrayMetadataJSON # noqa: TC001 +from zarr_metadata.v2.group import ZarrV2GroupMetadataJSON # noqa: TC001 +from zarr_metadata.v3.array import ZarrV3ArrayMetadataJSON # noqa: TC001 +from zarr_metadata.v3.group import ZarrV3GroupMetadataJSON # noqa: TC001 + +DocumentT = TypeVar("DocumentT") + + +@dataclass(frozen=True, slots=True) +class Valid(Generic[DocumentT]): + """A document that passed structural and composition validation.""" + + document: DocumentT + valid: Literal[True] = True + + +@dataclass(frozen=True, slots=True) +class Invalid: + """Every reason a document failed validation. + + `problems` is never empty: an empty report is a `Valid`. + """ + + problems: tuple[ValidationProblem, ...] + valid: Literal[False] = False + + def __post_init__(self) -> None: + if len(self.problems) == 0: + msg = "Invalid requires at least one validation problem" + raise ValueError(msg) + + +ValidationResult: TypeAlias = Valid[DocumentT] | Invalid +"""Either a validated document or the problems that disqualified it.""" + + +def check_array_metadata_v3(value: object) -> ValidationResult[ZarrV3ArrayMetadataJSON]: + """`value` as a valid v3 array document, or the problems disqualifying it. + + A `Valid` carries the normalized document (JSON arrays as tuples), + exactly as `parse_array_metadata_v3` returns it. + """ + problems = validate_array_metadata_v3(value) + if len(problems) != 0: + return Invalid(problems) + return Valid(cast("ZarrV3ArrayMetadataJSON", arrays_to_tuples(value))) + + +def check_array_metadata_v2(value: object) -> ValidationResult[ZarrV2ArrayMetadataJSON]: + """`value` as a valid v2 array document, or the problems disqualifying it.""" + problems = validate_array_metadata_v2(value) + if len(problems) != 0: + return Invalid(problems) + return Valid(cast("ZarrV2ArrayMetadataJSON", arrays_to_tuples(value))) + + +def check_group_metadata_v3(value: object) -> ValidationResult[ZarrV3GroupMetadataJSON]: + """`value` as a valid v3 group document, or the problems disqualifying it.""" + problems = validate_group_metadata_v3(value) + if len(problems) != 0: + return Invalid(problems) + return Valid(cast("ZarrV3GroupMetadataJSON", arrays_to_tuples(value))) + + +def check_group_metadata_v2(value: object) -> ValidationResult[ZarrV2GroupMetadataJSON]: + """`value` as a valid v2 group document, or the problems disqualifying it.""" + problems = validate_group_metadata_v2(value) + if len(problems) != 0: + return Invalid(problems) + return Valid(cast("ZarrV2GroupMetadataJSON", arrays_to_tuples(value))) + + +__all__ = [ + "Invalid", + "Valid", + "ValidationResult", + "check_array_metadata_v2", + "check_array_metadata_v3", + "check_group_metadata_v2", + "check_group_metadata_v3", +] diff --git a/packages/zarr-metadata/src/zarr_metadata/rules/_spec.py b/packages/zarr-metadata/src/zarr_metadata/rules/_spec.py new file mode 100644 index 0000000000..88459c26b4 --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/rules/_spec.py @@ -0,0 +1,156 @@ +"""Array specifications and how a codec chain transforms them. + +Each array->array codec transforms the array it receives, so a codec's +configuration must be judged against the array that *reaches* it, not +against the document's top-level fields: `transpose` permutes the shape, +`cast_value` changes the data type, and a `sharding_indexed` codec that +follows either one sees the transformed array. + +`ArraySpec` is the array a codec receives; `propagate` walks a chain +handing each codec its incoming spec. A field is `None` once this package +can no longer determine it. An unknown codec might change anything, so +every codec after one receives `NOTHING_KNOWN` and rules that need a +field decline rather than guess. Shape stops at the array->bytes +boundary; the data type carries through. + +Transitions are registered per array->array codec, next to that codec's +rules, via `spec_transition`. A modelled codec with no transition is +treated as unknown, so a forgotten transition fails closed. +""" + +from __future__ import annotations + +from collections.abc import Callable, Iterator, Mapping, Sequence +from dataclasses import dataclass, replace +from typing import TYPE_CHECKING, Final + +from zarr_metadata.v3._extension_points import CODECS, canonical_name +from zarr_metadata.v3._shape import entity_name +from zarr_metadata.v3.codec.kind import codec_kind_of_name + +if TYPE_CHECKING: + from zarr_metadata.v3._common import ZarrV3MetadataFieldJSON + + +@dataclass(frozen=True, slots=True) +class ArraySpec: + """The array a codec receives; a field is `None` when undetermined. + + `data_type` is the metadata-field value verbatim (a bare name or a + name/configuration object) because rules compare it by name. + """ + + shape: tuple[int, ...] | None + data_type: ZarrV3MetadataFieldJSON | None + + def with_shape(self, shape: tuple[int, ...] | None) -> ArraySpec: + return replace(self, shape=shape) + + def with_data_type(self, data_type: ZarrV3MetadataFieldJSON | None) -> ArraySpec: + return replace(self, data_type=data_type) + + +NOTHING_KNOWN: Final = ArraySpec(None, None) +"""The spec past a point where nothing about the array can be determined. + +Compare by equality: a spec can arrive here field by field and is then +equal to this constant without being it. +""" + + +SpecTransition = Callable[[Mapping[str, object], ArraySpec], ArraySpec] +"""How one codec transforms the spec it receives. + +Takes the codec's (shape-valid) configuration and the incoming spec, and +returns the outgoing one. A transition must never raise on the values the +shape validator admits; a field it cannot determine becomes `None`. +""" + +_TRANSITIONS: Final[dict[str, SpecTransition]] = {} + + +def spec_transition(codec: str) -> Callable[[SpecTransition], SpecTransition]: + """Register how `codec` transforms an incoming `ArraySpec`. + + Only array->array codecs need one: array->bytes and bytes->bytes + codecs end shape propagation by construction, so registering a + transition for one is refused. + """ + kind = codec_kind_of_name(codec) + if kind != "array_array": + msg = ( + f"spec transition registered for {codec!r}, which is " + f"{kind or 'unknown'} rather than array_array; only array->array " + "codecs transform the array spec" + ) + raise ValueError(msg) + + def decorate(transition: SpecTransition) -> SpecTransition: + _TRANSITIONS[canonical_name(CODECS, codec)] = transition + return transition + + return decorate + + +def transitions_registered() -> frozenset[str]: + """Every codec name with a registered spec transition.""" + return frozenset(_TRANSITIONS) + + +def propagate( + codecs: Sequence[object], + initial: ArraySpec, + configuration_of: Callable[[object], Mapping[str, object] | None], +) -> Iterator[tuple[int, object, ArraySpec]]: + """Yield `(index, codec, incoming_spec)` for each codec in the chain. + + `incoming_spec` is `NOTHING_KNOWN` once propagation has stopped: after + an unknown codec, after a known codec whose configuration is not + shape-valid, or after a codec this package has no transition for. + `configuration_of` resolves a codec entry to its usable configuration + (`entity_configuration` in practice; injected to keep this module free + of the registry). + """ + spec = initial + for index, codec in enumerate(codecs): + yield index, codec, spec + if spec == NOTHING_KNOWN: + continue + name = entity_name(codec) + kind = codec_kind_of_name(name) if name is not None else None + if kind is None: + spec = NOTHING_KNOWN + elif kind == "array_array": + transition = _TRANSITIONS.get(canonical_name(CODECS, name or "")) + configuration = configuration_of(codec) + if transition is None or configuration is None: + spec = NOTHING_KNOWN + else: + spec = transition(configuration, spec) + else: + # array->bytes: the array is gone; bytes->bytes: never had one. + spec = spec.with_shape(None) + + +def initial_spec(document: Mapping[str, object], chunk_shape: tuple[int, ...] | None) -> ArraySpec: + """The spec entering a document's top-level codec chain. + + The array a chunk pipeline encodes is one chunk, so the incoming shape + is the chunk grid's chunk shape (`None` if the grid is not a regular + grid this package can read). The data type is the document's own. + """ + data_type = document.get("data_type") + if not isinstance(data_type, (str, Mapping)): + data_type = None + return ArraySpec(chunk_shape, data_type) # type: ignore[arg-type] + + +__all__ = [ + "NOTHING_KNOWN", + "ArraySpec", + "SpecTransition", + "initial_spec", + "propagate", + "spec_transition", + "transitions_registered", +] diff --git a/packages/zarr-metadata/src/zarr_metadata/rules/_v2_array.py b/packages/zarr-metadata/src/zarr_metadata/rules/_v2_array.py new file mode 100644 index 0000000000..cd4493dc59 --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/rules/_v2_array.py @@ -0,0 +1,54 @@ +"""Composition rules for v2 array metadata documents. + +The v2 rule set is deliberately small today: the one cross-field +constraint the package interprets is that `chunks` and `shape` agree on +dimensionality. Fill-value/dtype consistency for v2 (NumPy dtype strings, +base64 fills for bytes dtypes) is a known follow-up, tracked in the +package docs. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Final + +from zarr_metadata.model._validation import ( + ARRAY_METADATA_STANDARD_KEYS_V2, + ValidationProblem, +) +from zarr_metadata.rules._engine import Rule, as_sequence +from zarr_metadata.rules._registry import document_rule, document_rules, register_document_type + +if TYPE_CHECKING: + from collections.abc import Mapping + + +ZARR_V2_ARRAY = "zarr_v2_array" +"""Document-type key under which this module's rules are registered.""" + +register_document_type(ZARR_V2_ARRAY, ARRAY_METADATA_STANDARD_KEYS_V2) + + +@document_rule(ZARR_V2_ARRAY, frozenset({"shape", "chunks"})) +def check_chunks_match_shape(document: Mapping[str, object]) -> tuple[ValidationProblem, ...]: + """`chunks` must have one entry per dimension of `shape`.""" + shape = as_sequence(document["shape"]) + chunks = as_sequence(document["chunks"]) + if shape is None or chunks is None or len(shape) == len(chunks): + return () + return ( + ValidationProblem( + ("chunks",), + "expected the same number of dimensions as shape", + "invalid_value", + ), + ) + + +ZARR_V2_ARRAY_RULES: Final[tuple[Rule, ...]] = document_rules(ZARR_V2_ARRAY) +"""The composition rule set for v2 array metadata documents.""" + + +__all__ = [ + "ZARR_V2_ARRAY", + "ZARR_V2_ARRAY_RULES", +] diff --git a/packages/zarr-metadata/src/zarr_metadata/rules/_v3_array.py b/packages/zarr-metadata/src/zarr_metadata/rules/_v3_array.py new file mode 100644 index 0000000000..1391707ddb --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/rules/_v3_array.py @@ -0,0 +1,428 @@ +"""Composition rules for v3 array metadata documents. + +Whole-document rules live here: judgments that read several top-level +fields, or that apply to a field regardless of which extension occupies +it. Rules about a *particular* codec or chunk grid live with that entity +in `zarr_metadata.rules._entities`, registered by name — so adding a +third chunk grid or a new codec adds a module there and changes nothing +in this one. The `codecs` and `chunk_grid` dispatchers below are generic: +they run whatever rules are registered for the name they find. + +Extension openness: rules never reject what they cannot interpret. An +unknown data type name accepts any fill value here (its own validator is +whoever understands it), an unknown codec has unknown kind, and unknown +entities pass through untouched. Openness is for genuinely unknown names +only: a codec or chunk-grid name this package defines is held to its full +canonical shape (via `zarr_metadata.v3._shape`), and a known codec ranks +as its pipeline kind in every spelling — otherwise a misspelled known +name would masquerade as an unknown extension and silently escape both +the shape and the ordering checks. +""" + +from __future__ import annotations + +from collections.abc import Mapping +from typing import TYPE_CHECKING, Final, cast + +from zarr_metadata.model._validation import ( + ARRAY_METADATA_STANDARD_KEYS_V3, + ValidationProblem, +) +from zarr_metadata.rules._engine import Rule, as_sequence, as_string_mapping, prefixed +from zarr_metadata.rules._pipeline import pipeline_order_problems, shape_problems +from zarr_metadata.rules._registry import ( + dispatch_field, + dispatch_field_sequence, + document_rule, + document_rules, + register_document_type, +) +from zarr_metadata.v3._extension_points import ( + CHUNK_GRID, + CHUNK_KEY_ENCODING, + CODECS, + DATA_TYPE, + ExtensionPointField, +) +from zarr_metadata.v3._shape import ( + validate_known_chunk_grid_metadata, + validate_known_entity_metadata, +) +from zarr_metadata.v3.data_type.bytes import base64_bytes +from zarr_metadata.v3.data_type.float16 import hex_float16 +from zarr_metadata.v3.data_type.float32 import hex_float32 +from zarr_metadata.v3.data_type.float64 import hex_float64 +from zarr_metadata.v3.data_type.raw import RAW_BYTES_NAME_PATTERN, raw_bytes_dtype_name + +if TYPE_CHECKING: + from collections.abc import Callable + + +# --------------------------------------------------------------------------- +# fill_value vs. data_type +# --------------------------------------------------------------------------- + +_INT_RANGES: Final[dict[str, tuple[int, int]]] = { + "int8": (-(2**7), 2**7 - 1), + "int16": (-(2**15), 2**15 - 1), + "int32": (-(2**31), 2**31 - 1), + "int64": (-(2**63), 2**63 - 1), + "uint8": (0, 2**8 - 1), + "uint16": (0, 2**16 - 1), + "uint32": (0, 2**32 - 1), + "uint64": (0, 2**64 - 1), +} + +_FLOAT_HEX_VALIDATORS: Final[dict[str, Callable[[str], object]]] = { + "float16": hex_float16, + "float32": hex_float32, + "float64": hex_float64, +} + +_COMPLEX_COMPONENT_TYPES: Final[dict[str, str]] = { + "complex64": "float32", + "complex128": "float64", +} + +_FLOAT_SPECIALS: Final = frozenset({"NaN", "Infinity", "-Infinity"}) + + +def _is_int(value: object) -> bool: + # bool is an int subtype but is never a valid integer fill value. + return isinstance(value, int) and not isinstance(value, bool) + + +def _check_float_fill(value: object, dtype_name: str) -> str | None: + if _is_int(value) or isinstance(value, float): + return None + if isinstance(value, str): + if value in _FLOAT_SPECIALS: + return None + try: + _FLOAT_HEX_VALIDATORS[dtype_name](value) + except ValueError: + return ( + f"expected a number, one of 'NaN'/'Infinity'/'-Infinity', or a " + f"{dtype_name} hex string, got {value!r}" + ) + return None + return f"expected a number or string, got {value!r}" + + +def _check_byte_sequence(value: object, expected_len: int | None) -> str | None: + items = as_sequence(value) + if items is None: + return f"expected an array of byte values, got {value!r}" + if expected_len is not None and len(items) != expected_len: + return f"expected {expected_len} byte values, got {len(items)}" + for item in items: + if not _is_int(item) or not 0 <= cast(int, item) <= 255: + return f"expected integers in [0, 255], got {item!r}" + return None + + +def _check_fill_for_dtype(dtype_name: str, value: object) -> str | None: + """Why `value` is not a valid fill value for `dtype_name`, or None. + + Unknown data type names accept anything (extension openness). + """ + if dtype_name == "bool": + return None if isinstance(value, bool) else f"expected a boolean, got {value!r}" + if dtype_name in _INT_RANGES: + low, high = _INT_RANGES[dtype_name] + if not _is_int(value): + return f"expected an integer, got {value!r}" + if not low <= cast(int, value) <= high: + return f"expected an integer in [{low}, {high}], got {value!r}" + return None + if dtype_name in _FLOAT_HEX_VALIDATORS: + return _check_float_fill(value, dtype_name) + if dtype_name in _COMPLEX_COMPONENT_TYPES: + component = _COMPLEX_COMPONENT_TYPES[dtype_name] + pair = as_sequence(value) + if pair is None or len(pair) != 2: + return f"expected a [real, imag] pair, got {value!r}" + for part in pair: + reason = _check_float_fill(part, component) + if reason is not None: + return f"invalid component: {reason}" + return None + if dtype_name == "string": + return None if isinstance(value, str) else f"expected a string, got {value!r}" + if dtype_name == "bytes": + if isinstance(value, str): + try: + base64_bytes(value) + except ValueError: + return f"expected standard-alphabet base64, got {value!r}" + return None + return _check_byte_sequence(value, None) + if dtype_name in ("numpy.datetime64", "numpy.timedelta64"): + if value == "NaT": + return None + if not _is_int(value): + return f"expected a signed 64-bit integer or 'NaT', got {value!r}" + if not -(2**63) <= cast(int, value) <= 2**63 - 1: + return f"expected a signed 64-bit integer, got {value!r}" + return None + if dtype_name == "struct": + if isinstance(value, Mapping): + return None + return f"expected an object of per-field fill values, got {value!r}" + if RAW_BYTES_NAME_PATTERN.fullmatch(dtype_name) is not None: + try: + raw_bytes_dtype_name(dtype_name) + except ValueError: + return None # malformed r name: _check_data_type_spelling reports it + return _check_byte_sequence(value, int(dtype_name[1:]) // 8) + return None # unknown data type: its fill values are not ours to judge + + +def _dtype_name(data_type: object) -> str | None: + if isinstance(data_type, str): + return data_type + mapping = as_string_mapping(data_type) + if mapping is not None: + name = mapping.get("name") + if isinstance(name, str): + return name + return None # structurally invalid; the structural validator reports it + + +def _check_data_type_spelling(document: Mapping[str, object]) -> tuple[ValidationProblem, ...]: + """Misspellings of data type families this package defines. + + An `r` name whose bit count is not a positive multiple of 8 is a + misspelling of the known raw-bytes family, not an unknown extension: + treating it as unknown would let the misspelling masquerade as an + extension and escape judgment entirely (the same anti-masquerade + reasoning as the codec spelling checks). Genuinely unknown names pass + untouched. + """ + name = _dtype_name(document["data_type"]) + if name is None or RAW_BYTES_NAME_PATTERN.fullmatch(name) is None: + return () + try: + raw_bytes_dtype_name(name) + except ValueError as error: + return (ValidationProblem(("data_type",), str(error), "invalid_value"),) + return () + + +def _check_fill_matches_dtype(document: Mapping[str, object]) -> tuple[ValidationProblem, ...]: + data_type = document["data_type"] + dtype_name = _dtype_name(data_type) + if dtype_name is None: + return () + if dtype_name == "struct": + return _struct_fill_problems(data_type, document["fill_value"], ("fill_value",)) + reason = _check_fill_for_dtype(dtype_name, document["fill_value"]) + if reason is None: + return () + return ( + ValidationProblem( + ("fill_value",), + f"fill_value invalid for data_type {dtype_name!r}: {reason}", + "invalid_value", + ), + ) + + +def _struct_fill_problems( + data_type: object, fill_value: object, loc: tuple[str | int, ...] +) -> tuple[ValidationProblem, ...]: + """Validate a struct fill mapping against every field, recursively.""" + if not isinstance(fill_value, Mapping): + return ( + ValidationProblem( + loc, + f"fill_value invalid for data_type 'struct': expected an object of " + f"per-field fill values, got {fill_value!r}", + "invalid_value", + ), + ) + envelope = as_string_mapping(data_type) + configuration = ( + as_string_mapping(envelope.get("configuration")) if envelope is not None else None + ) + fields = configuration.get("fields") if configuration is not None else None + if not isinstance(fields, tuple): + return () # malformed data type: the structural validator owns it + + fill_mapping = cast("Mapping[object, object]", fill_value) + problems: list[ValidationProblem] = [] + field_names: set[str] = set() + for field in cast("tuple[object, ...]", fields): + field_mapping = as_string_mapping(field) + if field_mapping is None: + continue + name = field_mapping.get("name") + field_data_type = field_mapping.get("data_type") + if not isinstance(name, str) or field_data_type is None: + continue + field_names.add(name) + field_loc = (*loc, name) + if name not in fill_mapping: + problems.append( + ValidationProblem( + field_loc, f"missing fill value for struct field {name!r}", "missing_key" + ) + ) + continue + field_fill = fill_mapping[name] + nested_name = _dtype_name(field_data_type) + if nested_name == "struct": + problems.extend(_struct_fill_problems(field_data_type, field_fill, field_loc)) + continue + if nested_name is None: + continue + reason = _check_fill_for_dtype(nested_name, field_fill) + if reason is not None: + problems.append( + ValidationProblem( + field_loc, + f"fill value invalid for struct field {name!r} with data_type " + f"{nested_name!r}: {reason}", + "invalid_value", + ) + ) + problems.extend( + ValidationProblem((*loc, key), f"unknown struct fill field {key!r}", "unknown_key") + for key in sorted( + candidate + for candidate in fill_mapping.keys() - field_names + if isinstance(candidate, str) + ) + ) + return tuple(problems) + + +# --------------------------------------------------------------------------- +# whole-document rules +# --------------------------------------------------------------------------- + +ZARR_V3_ARRAY = "zarr_v3_array" +"""Document-type key under which this module's rules are registered.""" + +register_document_type(ZARR_V3_ARRAY, ARRAY_METADATA_STANDARD_KEYS_V3) + +_data_type_spelling = document_rule(ZARR_V3_ARRAY, frozenset({"data_type"}))( + _check_data_type_spelling +) +_fill_matches_dtype = document_rule(ZARR_V3_ARRAY, frozenset({"data_type", "fill_value"}))( + _check_fill_matches_dtype +) + + +def _known_entity_shape( + field: ExtensionPointField, +) -> Callable[[Mapping[str, object]], tuple[ValidationProblem, ...]]: + """A check that judges `document[field]` against `field`'s known shapes. + + One parameter, not two: the document field and the extension point are + the same thing, and taking them separately invited passing a codec + under the chunk-grid point. + """ + + def check(document: Mapping[str, object]) -> tuple[ValidationProblem, ...]: + found = validate_known_entity_metadata(field, document[field]) + return () if found is None else prefixed((field,), found) + + return check + + +_data_type_shape = document_rule(ZARR_V3_ARRAY, frozenset({"data_type"}))( + _known_entity_shape(DATA_TYPE) +) +_chunk_key_encoding_shape = document_rule(ZARR_V3_ARRAY, frozenset({"chunk_key_encoding"}))( + _known_entity_shape(CHUNK_KEY_ENCODING) +) + + +@document_rule(ZARR_V3_ARRAY, frozenset({"codecs"})) +def check_codec_pipeline_order(document: Mapping[str, object]) -> tuple[ValidationProblem, ...]: + """The pipeline shape: `array->array`* `array->bytes` `bytes->bytes`*.""" + entries = as_sequence(document["codecs"]) + if entries is None: + return () + return pipeline_order_problems(entries, ("codecs",)) + + +@document_rule(ZARR_V3_ARRAY, frozenset({"codecs"})) +def check_codec_shapes(document: Mapping[str, object]) -> tuple[ValidationProblem, ...]: + """Every known-name codec matches its canonical type.""" + entries = as_sequence(document["codecs"]) + if entries is None: + return () + return shape_problems(entries, ("codecs",)) + + +@document_rule(ZARR_V3_ARRAY, frozenset({"chunk_grid"})) +def check_chunk_grid_shape(document: Mapping[str, object]) -> tuple[ValidationProblem, ...]: + """A known-name chunk grid matches its canonical type.""" + found = validate_known_chunk_grid_metadata(document["chunk_grid"]) + # None is "not a known grid" (unjudged); () is "known and valid". + if found is None: + return () + return prefixed(("chunk_grid",), found) + + +@document_rule(ZARR_V3_ARRAY, frozenset({"shape", "dimension_names"})) +def check_dimension_names_length(document: Mapping[str, object]) -> tuple[ValidationProblem, ...]: + """One dimension name per array dimension.""" + shape = as_sequence(document["shape"]) + names = as_sequence(document["dimension_names"]) + if shape is None or names is None or len(names) == len(shape): + return () + return ( + ValidationProblem( + ("dimension_names",), + f"dimension_names has {len(names)} entries but shape has {len(shape)} dimensions", + "invalid_value", + ), + ) + + +# Generic dispatchers: every rule an entity registers for itself runs here, +# so a new codec, chunk grid, data type, or chunk key encoding needs no edit +# to this module. There must be one per extension point that has shapes: +# without it, `entity_rule` accepts a registration whose rule can never run, +# which is the silent-pass failure the registry exists to prevent. +# `test_registry.py` asserts that coverage. +_dispatch_chunk_grid = document_rule(ZARR_V3_ARRAY, frozenset({"chunk_grid"}))( + dispatch_field(CHUNK_GRID) +) +_dispatch_data_type = document_rule(ZARR_V3_ARRAY, frozenset({"data_type"}))( + dispatch_field(DATA_TYPE) +) +_dispatch_chunk_key_encoding = document_rule(ZARR_V3_ARRAY, frozenset({"chunk_key_encoding"}))( + dispatch_field(CHUNK_KEY_ENCODING) +) +_dispatch_codecs = document_rule(ZARR_V3_ARRAY, frozenset({"codecs"}))( + dispatch_field_sequence(CODECS) +) + + +def _rules() -> tuple[Rule, ...]: + # Importing the entity package registers every entity's rules; done here + # rather than at module import to keep the dependency one-directional. + import zarr_metadata.rules._entities as entity_rules_package + + # Imported for its registrations; referenced so the import cannot be + # pruned as unused by a checker or a well-meaning cleanup. + assert entity_rules_package is not None + return document_rules(ZARR_V3_ARRAY) + + +ZARR_V3_ARRAY_RULES: Final[tuple[Rule, ...]] = _rules() +"""The composition rule set for v3 array metadata documents. + +Assembled from the registry rather than written out, so a rule cannot be +defined without joining the set it belongs to. +""" + + +__all__ = [ + "ZARR_V3_ARRAY", + "ZARR_V3_ARRAY_RULES", +] diff --git a/packages/zarr-metadata/src/zarr_metadata/rules/_v3_group.py b/packages/zarr-metadata/src/zarr_metadata/rules/_v3_group.py new file mode 100644 index 0000000000..5023ef11bd --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/rules/_v3_group.py @@ -0,0 +1,85 @@ +"""Composition rules for v3 group metadata documents. + +A group document's own fields carry no cross-field constraints, but the +inline consolidated-metadata convention embeds whole child documents — +and a composition-invalid child makes the consolidated view lie about +the store. The group rule set therefore recurses: every array entry is +judged by the v3 array rules, and every group entry (which may itself +carry consolidated metadata) by this rule set. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Final + +from zarr_metadata.model._validation import GROUP_METADATA_STANDARD_KEYS_V3 +from zarr_metadata.rules._engine import Rule, as_string_mapping, prefixed, run_rules +from zarr_metadata.rules._registry import document_rule, document_rules, register_document_type +from zarr_metadata.rules._v3_array import ZARR_V3_ARRAY_RULES +from zarr_metadata.v3.consolidated import ZARR_V3_CONSOLIDATED_METADATA_KEY + +if TYPE_CHECKING: + from collections.abc import Mapping + + from zarr_metadata.model._validation import ValidationProblem + + +ZARR_V3_GROUP = "zarr_v3_group" +"""Document-type key under which this module's rules are registered.""" + +# `consolidated_metadata` is not a declared member of the group TypedDict: +# the spec grandfathers it as a convention that 'lacks the name member +# required of extension objects'. It is declared here so the rule that +# reads it passes the typo check without exempting unknown keys. +register_document_type( + ZARR_V3_GROUP, + GROUP_METADATA_STANDARD_KEYS_V3, + extension_keys=frozenset({ZARR_V3_CONSOLIDATED_METADATA_KEY}), +) + + +@document_rule(ZARR_V3_GROUP, frozenset({"consolidated_metadata"})) +def check_consolidated_entries(document: Mapping[str, object]) -> tuple[ValidationProblem, ...]: + """Consolidated child documents must satisfy their own composition rules. + + Structural validity of the consolidated envelope and its entries is + the model layer's job; entries that are not interpretable as node + documents decline in its favor. + """ + return consolidated_entries_problems( + document["consolidated_metadata"], ("consolidated_metadata",) + ) + + +def consolidated_entries_problems( + value: object, loc: tuple[str | int, ...] = () +) -> tuple[ValidationProblem, ...]: + """Composition problems in an inline consolidated envelope's children.""" + consolidated = as_string_mapping(value) + if consolidated is None: + return () + metadata = as_string_mapping(consolidated.get("metadata")) + if metadata is None: + return () + problems: list[ValidationProblem] = [] + for path, entry in metadata.items(): + node = as_string_mapping(entry) + if node is None: + continue + entry_loc = (*loc, "metadata", path) + node_type = node.get("node_type") + if node_type == "array": + problems.extend(prefixed(entry_loc, run_rules(ZARR_V3_ARRAY_RULES, node))) + elif node_type == "group": + problems.extend(prefixed(entry_loc, run_rules(ZARR_V3_GROUP_RULES, node))) + return tuple(problems) + + +ZARR_V3_GROUP_RULES: Final[tuple[Rule, ...]] = document_rules(ZARR_V3_GROUP) +"""The composition rule set for v3 group metadata documents.""" + + +__all__ = [ + "ZARR_V3_GROUP", + "ZARR_V3_GROUP_RULES", +] diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/_extension_points.py b/packages/zarr-metadata/src/zarr_metadata/v3/_extension_points.py new file mode 100644 index 0000000000..03a8472a2b --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/v3/_extension_points.py @@ -0,0 +1,54 @@ +"""The Zarr v3 extension points, and how names are keyed under them. + +Names are unique only within an extension point (`bytes` is both a core +codec and a registered data type), so every table in this package is +keyed by `(field, canonical name)`. + +`canonical_name` is identity except for raw-byte data types: every `r` +spelling, valid or not, maps to `RAW_BYTES_FAMILY`, so a malformed member +of that family is reported as a misspelling rather than passing as an +unknown extension. Canonical names are lookup keys and are never emitted. +""" + +from __future__ import annotations + +from typing import Final, Literal + +from zarr_metadata.v3.data_type.raw import RAW_BYTES_NAME_PATTERN + +ExtensionPointField = Literal[ + "data_type", "chunk_grid", "chunk_key_encoding", "codecs", "storage_transformers" +] +"""The v3 array metadata fields whose values name an extension.""" + +DATA_TYPE: Final[ExtensionPointField] = "data_type" +CHUNK_GRID: Final[ExtensionPointField] = "chunk_grid" +CHUNK_KEY_ENCODING: Final[ExtensionPointField] = "chunk_key_encoding" +CODECS: Final[ExtensionPointField] = "codecs" +STORAGE_TRANSFORMERS: Final[ExtensionPointField] = "storage_transformers" + +RAW_BYTES_FAMILY: Final = "r" +"""Canonical key for the parameterized raw-bytes data type family. + +Spelled as the spec writes the family; the angle brackets keep it +unforgeable by a real name. +""" + + +def canonical_name(field: ExtensionPointField, name: str) -> str: + """`name` reduced to the key this package tables it under.""" + if field == DATA_TYPE and RAW_BYTES_NAME_PATTERN.fullmatch(name) is not None: + return RAW_BYTES_FAMILY + return name + + +__all__ = [ + "CHUNK_GRID", + "CHUNK_KEY_ENCODING", + "CODECS", + "DATA_TYPE", + "RAW_BYTES_FAMILY", + "STORAGE_TRANSFORMERS", + "ExtensionPointField", + "canonical_name", +] diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/_shape.py b/packages/zarr-metadata/src/zarr_metadata/v3/_shape.py new file mode 100644 index 0000000000..a31cc04d52 --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/v3/_shape.py @@ -0,0 +1,666 @@ +""" +Type-level shape validation for known metadata-field entities. + +One validator per extension-point entity this package defines, exact with +respect to the entity's declared TypedDicts: a value yields no problems +exactly when it is an instance of the canonical metadata type, so a +verdict here means the same thing the type does. + +Two package-wide conventions qualify "exact": + +- `int`-annotated fields mean JSON integers, so JSON booleans are + rejected even though `bool` is an `int` subtype in Python's type + system (matching the fill-value rules' treatment of integers). +- Judgments are at the canonical data level: JSON arrays are tuples, as + the TypedDicts declare. Normalize a freshly-`json.loads`-ed document + (e.g. with a model-layer parser) before asking for shape verdicts. + +Value judgments beyond the types — permutation contents, shard geometry, +cross-field consistency — belong to the composition rule layer, not here. + +Unknown names are not judged (extension openness): the `validate_known_*` +functions answer `None` for entities this package has no types for, no +problems for a valid known entity, and problems otherwise. + +Key sets are derived from the TypedDicts' `__annotations__` / +`__required_keys__` rather than restated by hand, so those entries cannot +drift from the canonical types; only the per-field value checks are +written out. The exception is `_BARE_DATA_TYPE_NAMES`: the core scalar +data types have no TypedDict to derive from — their whole metadata is a +name — so that list is hand-written, and `tests/test_registry_drift.py` +ties it to the modules that define those names. +""" + +from __future__ import annotations + +from collections.abc import Mapping, Sequence +from dataclasses import dataclass +from typing import TYPE_CHECKING, Final, cast + +from zarr_metadata.model._validation import ( + ValidationProblem, + is_json, + is_metadata_field_v3, +) +from zarr_metadata.v3._extension_points import ( + CHUNK_GRID, + CHUNK_KEY_ENCODING, + CODECS, + DATA_TYPE, + RAW_BYTES_FAMILY, + ExtensionPointField, + canonical_name, +) +from zarr_metadata.v3.chunk_grid.rectilinear import ( + RECTILINEAR_CHUNK_GRID_NAME, + RectilinearChunkGridConfiguration, + RectilinearChunkGridObject, +) +from zarr_metadata.v3.chunk_grid.regular import ( + REGULAR_CHUNK_GRID_NAME, + RegularChunkGridConfiguration, + RegularChunkGridObject, +) +from zarr_metadata.v3.chunk_key_encoding.default import ( + DEFAULT_CHUNK_KEY_ENCODING_NAME, + DEFAULT_CHUNK_KEY_ENCODING_SEPARATOR, + DefaultChunkKeyEncodingConfiguration, + DefaultChunkKeyEncodingObject, +) +from zarr_metadata.v3.chunk_key_encoding.v2 import ( + V2_CHUNK_KEY_ENCODING_NAME, + V2_CHUNK_KEY_ENCODING_SEPARATOR, + V2ChunkKeyEncodingConfiguration, + V2ChunkKeyEncodingObject, +) +from zarr_metadata.v3.codec.blosc import ( + BLOSC_CNAME, + BLOSC_CODEC_NAME, + BLOSC_SHUFFLE, + BloscCodecConfiguration, + BloscCodecObject, +) +from zarr_metadata.v3.codec.bytes import ( + BYTES_CODEC_NAME, + ENDIANNESS, + BytesCodecConfiguration, + BytesCodecObject, +) +from zarr_metadata.v3.codec.cast_value import ( + CAST_OUT_OF_RANGE_MODE, + CAST_ROUNDING_MODE, + CAST_VALUE_CODEC_NAME, + CastValueCodecConfiguration, + CastValueCodecObject, + ScalarMap, +) +from zarr_metadata.v3.codec.crc32c import CRC32C_CODEC_NAME, Crc32cCodecObject, Empty +from zarr_metadata.v3.codec.gzip import GZIP_CODEC_NAME, GzipCodecConfiguration, GzipCodecObject +from zarr_metadata.v3.codec.scale_offset import ( + SCALE_OFFSET_CODEC_NAME, + ScaleOffsetCodecConfiguration, + ScaleOffsetCodecObject, +) +from zarr_metadata.v3.codec.sharding_indexed import ( + SHARDING_INDEX_LOCATION, + SHARDING_INDEXED_CODEC_NAME, + ShardingIndexedCodecConfiguration, + ShardingIndexedCodecObject, +) +from zarr_metadata.v3.codec.transpose import ( + TRANSPOSE_CODEC_NAME, + TransposeCodecConfiguration, + TransposeCodecObject, +) +from zarr_metadata.v3.codec.zstd import ZSTD_CODEC_NAME, ZstdCodecConfiguration, ZstdCodecObject +from zarr_metadata.v3.data_type.bool import BOOL_DATA_TYPE_NAME +from zarr_metadata.v3.data_type.bytes import BYTES_DATA_TYPE_NAME +from zarr_metadata.v3.data_type.complex64 import COMPLEX64_DATA_TYPE_NAME +from zarr_metadata.v3.data_type.complex128 import COMPLEX128_DATA_TYPE_NAME +from zarr_metadata.v3.data_type.float16 import FLOAT16_DATA_TYPE_NAME +from zarr_metadata.v3.data_type.float32 import FLOAT32_DATA_TYPE_NAME +from zarr_metadata.v3.data_type.float64 import FLOAT64_DATA_TYPE_NAME +from zarr_metadata.v3.data_type.int8 import INT8_DATA_TYPE_NAME +from zarr_metadata.v3.data_type.int16 import INT16_DATA_TYPE_NAME +from zarr_metadata.v3.data_type.int32 import INT32_DATA_TYPE_NAME +from zarr_metadata.v3.data_type.int64 import INT64_DATA_TYPE_NAME +from zarr_metadata.v3.data_type.numpy_datetime64 import ( + NUMPY_DATETIME64_DATA_TYPE_NAME, + NumpyDatetime64, + NumpyDatetime64Configuration, +) +from zarr_metadata.v3.data_type.numpy_timedelta64 import ( + NUMPY_TIME_UNIT, + NUMPY_TIMEDELTA64_DATA_TYPE_NAME, + NumpyTimedelta64, + NumpyTimedelta64Configuration, +) +from zarr_metadata.v3.data_type.string import STRING_DATA_TYPE_NAME +from zarr_metadata.v3.data_type.struct import ( + STRUCT_DATA_TYPE_NAME, + Struct, + StructConfiguration, + StructField, +) +from zarr_metadata.v3.data_type.uint8 import UINT8_DATA_TYPE_NAME +from zarr_metadata.v3.data_type.uint16 import UINT16_DATA_TYPE_NAME +from zarr_metadata.v3.data_type.uint32 import UINT32_DATA_TYPE_NAME +from zarr_metadata.v3.data_type.uint64 import UINT64_DATA_TYPE_NAME + +if TYPE_CHECKING: + from collections.abc import Callable + + from zarr_metadata.model._validation import ProblemKind + + _FieldChecker = Callable[[object, tuple[str | int, ...]], tuple[ValidationProblem, ...]] + + +def entity_name(value: object) -> str | None: + """The `name` of a metadata-field entry in any spelling, or None. + + A bare string is its own name; an object's name is its `name` member. + Anything else (or a mapping without a string `name`) has no name and + is not interpretable as a known entity. + """ + if isinstance(value, str): + return value + if not isinstance(value, Mapping): + return None + name = cast("Mapping[object, object]", value).get("name") + return name if isinstance(name, str) else None + + +def _problems( + loc: tuple[str | int, ...], message: str, kind: ProblemKind = "invalid_type" +) -> tuple[ValidationProblem, ...]: + return (ValidationProblem(loc, message, kind),) + + +def _check_json_int(value: object, loc: tuple[str | int, ...]) -> tuple[ValidationProblem, ...]: + if isinstance(value, bool) or not isinstance(value, int): + return _problems(loc, f"expected an integer, got {value!r}") + return () + + +def _check_json_bool(value: object, loc: tuple[str | int, ...]) -> tuple[ValidationProblem, ...]: + if not isinstance(value, bool): + return _problems(loc, f"expected a boolean, got {value!r}") + return () + + +def _literal(allowed: tuple[str, ...]) -> _FieldChecker: + def check(value: object, loc: tuple[str | int, ...]) -> tuple[ValidationProblem, ...]: + if value not in allowed: + return _problems(loc, f"expected one of {allowed!r}, got {value!r}", "invalid_value") + return () + + return check + + +def _check_int_tuple(value: object, loc: tuple[str | int, ...]) -> tuple[ValidationProblem, ...]: + if not isinstance(value, tuple): + return _problems(loc, f"expected an array (tuple) of integers, got {value!r}") + items = cast("tuple[object, ...]", value) + return tuple( + problem + for index, item in enumerate(items) + for problem in _check_json_int(item, (*loc, index)) + ) + + +def _check_json_value(value: object, loc: tuple[str | int, ...]) -> tuple[ValidationProblem, ...]: + if not is_json(value): + return _problems(loc, "expected a JSON value") + return () + + +def _check_metadata_field( + value: object, loc: tuple[str | int, ...] +) -> tuple[ValidationProblem, ...]: + if not is_metadata_field_v3(value): + return _problems(loc, "expected a metadata field (bare name or name/configuration object)") + return () + + +def _check_field_tuple(value: object, loc: tuple[str | int, ...]) -> tuple[ValidationProblem, ...]: + if not isinstance(value, tuple): + return _problems(loc, f"expected an array (tuple) of metadata fields, got {value!r}") + items = cast("tuple[object, ...]", value) + return tuple( + problem + for index, item in enumerate(items) + for problem in _check_metadata_field(item, (*loc, index)) + ) + + +_STRUCT_FIELD_KEYS: Final = frozenset(StructField.__annotations__) + + +def _check_data_type_field( + value: object, loc: tuple[str | int, ...] +) -> tuple[ValidationProblem, ...]: + """A nested `data_type` position: structural shape plus its known shape. + + `cast_value`'s target type and a struct field's type are data types + like any other, so they get the judgment a top-level `data_type` gets. + Without this the same value is accepted in one position and rejected + in another — a bare `"numpy.datetime64"` is invalid at the top level + (its configuration is required) and was silently fine inside a struct. + Recurses naturally: a struct of structs is judged all the way down. + """ + problems = _check_metadata_field(value, loc) + if len(problems) != 0: + return problems + found = validate_known_entity_metadata(DATA_TYPE, value) + return () if found is None else _prefixed_at(loc, found) + + +def _prefixed_at( + loc: tuple[str | int, ...], problems: tuple[ValidationProblem, ...] +) -> tuple[ValidationProblem, ...]: + return tuple( + ValidationProblem((*loc, *problem.loc), problem.message, problem.kind) + for problem in problems + ) + + +def _check_struct_fields( + value: object, loc: tuple[str | int, ...] +) -> tuple[ValidationProblem, ...]: + if not isinstance(value, tuple): + return _problems(loc, f"expected an array (tuple) of struct fields, got {value!r}") + problems: list[ValidationProblem] = [] + for index, item in enumerate(cast("tuple[object, ...]", value)): + item_loc = (*loc, index) + if not isinstance(item, Mapping): + problems.extend(_problems(item_loc, f"expected an object, got {item!r}")) + continue + field = cast("Mapping[object, object]", item) + for key in field: + if not isinstance(key, str) or key not in _STRUCT_FIELD_KEYS: + problems.extend(_problems(item_loc, f"unexpected key {key!r}", "unknown_key")) + for key in sorted(_STRUCT_FIELD_KEYS - field.keys()): + problems.extend(_problems((*item_loc, key), "missing required key", "missing_key")) + if "name" in field and not isinstance(field["name"], str): + problems.extend(_problems((*item_loc, "name"), "expected a string")) + if "data_type" in field: + problems.extend(_check_data_type_field(field["data_type"], (*item_loc, "data_type"))) + return tuple(problems) + + +_SCALAR_MAP_KEYS: Final = frozenset(ScalarMap.__annotations__) + + +def _check_scalar_map_entries( + value: object, loc: tuple[str | int, ...] +) -> tuple[ValidationProblem, ...]: + if not isinstance(value, tuple): + return _problems(loc, f"expected an array (tuple) of [old, new] pairs, got {value!r}") + problems: list[ValidationProblem] = [] + for index, item in enumerate(cast("tuple[object, ...]", value)): + if not isinstance(item, tuple) or len(cast("tuple[object, ...]", item)) != 2: + problems.extend(_problems((*loc, index), f"expected an [old, new] pair, got {item!r}")) + continue + for position, scalar in enumerate(cast("tuple[object, ...]", item)): + problems.extend(_check_json_value(scalar, (*loc, index, position))) + return tuple(problems) + + +def _check_scalar_map(value: object, loc: tuple[str | int, ...]) -> tuple[ValidationProblem, ...]: + if not isinstance(value, Mapping): + return _problems(loc, f"expected an object, got {value!r}") + mapping = cast("Mapping[object, object]", value) + problems: list[ValidationProblem] = [] + for key in mapping: + if not isinstance(key, str) or key not in _SCALAR_MAP_KEYS: + problems.extend(_problems((*loc,), f"unexpected key {key!r}", "unknown_key")) + for key in _SCALAR_MAP_KEYS: + if key in mapping: + problems.extend(_check_scalar_map_entries(mapping[key], (*loc, key))) + return tuple(problems) + + +def _check_rectilinear_dim_spec( + value: object, loc: tuple[str | int, ...] +) -> tuple[ValidationProblem, ...]: + if not isinstance(value, bool) and isinstance(value, int): + return () + if not isinstance(value, tuple): + return _problems( + loc, + f"expected an integer or an array of integers / [value, count] pairs, got {value!r}", + ) + problems: list[ValidationProblem] = [] + for index, item in enumerate(cast("tuple[object, ...]", value)): + if not isinstance(item, bool) and isinstance(item, int): + continue + if isinstance(item, tuple) and len(cast("tuple[object, ...]", item)) == 2: + problems.extend( + problem + for position, part in enumerate(cast("tuple[object, ...]", item)) + for problem in _check_json_int(part, (*loc, index, position)) + ) + continue + problems.extend( + _problems((*loc, index), f"expected an integer or a [value, count] pair, got {item!r}") + ) + return tuple(problems) + + +def _check_rectilinear_dim_specs( + value: object, loc: tuple[str | int, ...] +) -> tuple[ValidationProblem, ...]: + if not isinstance(value, tuple): + return _problems(loc, f"expected an array (tuple) of dimension specs, got {value!r}") + return tuple( + problem + for index, item in enumerate(cast("tuple[object, ...]", value)) + for problem in _check_rectilinear_dim_spec(item, (*loc, index)) + ) + + +@dataclass(frozen=True, slots=True) +class _EntityShape: + """Shape facts for one known entity name, derived from its TypedDicts.""" + + object_keys: frozenset[str] + configuration_required: bool + config_keys: frozenset[str] + config_required: frozenset[str] + config_checkers: Mapping[str, _FieldChecker] + + +def _shape( + object_type: type, + configuration_type: type, + checkers: Mapping[str, _FieldChecker], +) -> _EntityShape: + config_keys = frozenset(configuration_type.__annotations__) + if frozenset(checkers) != config_keys: + raise AssertionError( # pragma: no cover - registry construction guard + f"checkers {sorted(checkers)} do not cover configuration keys {sorted(config_keys)}" + ) + return _EntityShape( + object_keys=frozenset(object_type.__annotations__), + configuration_required="configuration" + in cast("frozenset[str]", object_type.__required_keys__), # type: ignore[attr-defined] + config_keys=config_keys, + config_required=cast( + "frozenset[str]", + configuration_type.__required_keys__, # type: ignore[attr-defined] + ), + config_checkers=dict(checkers), + ) + + +def _bare_shape() -> _EntityShape: + """Shape of an entity that takes no configuration. + + Both spellings are valid: the spec makes `{"name": ...}` the base form + and permits the bare short-hand when no configuration is required. The + object form is therefore accepted with an absent or empty + `configuration`, and any member inside one is an unknown key. + + Keyword arguments deliberately: this is a six-field record whose flags + are easy to transpose positionally. + """ + return _EntityShape( + object_keys=frozenset({"name", "configuration", "must_understand"}), + configuration_required=False, + config_keys=frozenset(), + config_required=frozenset(), + config_checkers={}, + ) + + +_CODEC_SHAPES: Final[Mapping[str, _EntityShape]] = { + BLOSC_CODEC_NAME: _shape( + BloscCodecObject, + BloscCodecConfiguration, + { + "cname": _literal(BLOSC_CNAME), + "clevel": _check_json_int, + "shuffle": _literal(BLOSC_SHUFFLE), + "blocksize": _check_json_int, + "typesize": _check_json_int, + }, + ), + BYTES_CODEC_NAME: _shape( + BytesCodecObject, BytesCodecConfiguration, {"endian": _literal(ENDIANNESS)} + ), + CAST_VALUE_CODEC_NAME: _shape( + CastValueCodecObject, + CastValueCodecConfiguration, + { + "data_type": _check_data_type_field, + "rounding": _literal(CAST_ROUNDING_MODE), + "out_of_range": _literal(CAST_OUT_OF_RANGE_MODE), + "scalar_map": _check_scalar_map, + }, + ), + CRC32C_CODEC_NAME: _shape(Crc32cCodecObject, Empty, {}), + GZIP_CODEC_NAME: _shape(GzipCodecObject, GzipCodecConfiguration, {"level": _check_json_int}), + SCALE_OFFSET_CODEC_NAME: _shape( + ScaleOffsetCodecObject, + ScaleOffsetCodecConfiguration, + {"offset": _check_json_value, "scale": _check_json_value}, + ), + SHARDING_INDEXED_CODEC_NAME: _shape( + ShardingIndexedCodecObject, + ShardingIndexedCodecConfiguration, + { + "chunk_shape": _check_int_tuple, + "codecs": _check_field_tuple, + "index_codecs": _check_field_tuple, + "index_location": _literal(SHARDING_INDEX_LOCATION), + }, + ), + TRANSPOSE_CODEC_NAME: _shape( + TransposeCodecObject, TransposeCodecConfiguration, {"order": _check_int_tuple} + ), + ZSTD_CODEC_NAME: _shape( + ZstdCodecObject, + ZstdCodecConfiguration, + {"level": _check_json_int, "checksum": _check_json_bool}, + ), +} + +_CHUNK_GRID_SHAPES: Final[Mapping[str, _EntityShape]] = { + REGULAR_CHUNK_GRID_NAME: _shape( + RegularChunkGridObject, RegularChunkGridConfiguration, {"chunk_shape": _check_int_tuple} + ), + RECTILINEAR_CHUNK_GRID_NAME: _shape( + RectilinearChunkGridObject, + RectilinearChunkGridConfiguration, + { + # "inline" is the sole member of the kind Literal; the type + # exports no constant tuple for it. + "kind": _literal(("inline",)), + "chunk_shapes": _check_rectilinear_dim_specs, + }, + ), +} + +_CHUNK_KEY_ENCODING_SHAPES: Final[Mapping[str, _EntityShape]] = { + DEFAULT_CHUNK_KEY_ENCODING_NAME: _shape( + DefaultChunkKeyEncodingObject, + DefaultChunkKeyEncodingConfiguration, + {"separator": _literal(DEFAULT_CHUNK_KEY_ENCODING_SEPARATOR)}, + ), + V2_CHUNK_KEY_ENCODING_NAME: _shape( + V2ChunkKeyEncodingObject, + V2ChunkKeyEncodingConfiguration, + {"separator": _literal(V2_CHUNK_KEY_ENCODING_SEPARATOR)}, + ), +} + +_BARE_DATA_TYPE_NAMES: Final = ( + BOOL_DATA_TYPE_NAME, + INT8_DATA_TYPE_NAME, + INT16_DATA_TYPE_NAME, + INT32_DATA_TYPE_NAME, + INT64_DATA_TYPE_NAME, + UINT8_DATA_TYPE_NAME, + UINT16_DATA_TYPE_NAME, + UINT32_DATA_TYPE_NAME, + UINT64_DATA_TYPE_NAME, + FLOAT16_DATA_TYPE_NAME, + FLOAT32_DATA_TYPE_NAME, + FLOAT64_DATA_TYPE_NAME, + COMPLEX64_DATA_TYPE_NAME, + COMPLEX128_DATA_TYPE_NAME, + RAW_BYTES_FAMILY, + BYTES_DATA_TYPE_NAME, + STRING_DATA_TYPE_NAME, +) + +_DATA_TYPE_SHAPES: Final[Mapping[str, _EntityShape]] = { + **{name: _bare_shape() for name in _BARE_DATA_TYPE_NAMES}, + NUMPY_DATETIME64_DATA_TYPE_NAME: _shape( + NumpyDatetime64, + NumpyDatetime64Configuration, + {"unit": _literal(NUMPY_TIME_UNIT), "scale_factor": _check_json_int}, + ), + NUMPY_TIMEDELTA64_DATA_TYPE_NAME: _shape( + NumpyTimedelta64, + NumpyTimedelta64Configuration, + {"unit": _literal(NUMPY_TIME_UNIT), "scale_factor": _check_json_int}, + ), + STRUCT_DATA_TYPE_NAME: _shape(Struct, StructConfiguration, {"fields": _check_struct_fields}), +} + + +def _validate_known_entity( + value: object, name: str, shape: _EntityShape, entity: str +) -> tuple[ValidationProblem, ...]: + """Every reason `value` is not an instance of `name`'s canonical type. + + Locations are relative to the entry itself (`("configuration", key)` + etc.); callers prefix the entry's position in its document. + """ + if isinstance(value, str): + if not shape.configuration_required: + return () + return _problems( + (), + f"{entity} {name!r} requires a configuration and has no bare short-hand " + f"form; use {{'name': {name!r}, 'configuration': {{...}}}}", + "invalid_value", + ) + if not isinstance(value, Mapping): + return _problems((), f"expected a bare name or an object, got {value!r}") + mapping = cast("Mapping[object, object]", value) + problems: list[ValidationProblem] = [] + for key in mapping: + if not isinstance(key, str) or key not in shape.object_keys: + problems.extend(_problems((), f"unexpected key {key!r}", "unknown_key")) + if "must_understand" in mapping: + problems.extend(_check_json_bool(mapping["must_understand"], ("must_understand",))) + if "configuration" not in mapping: + if shape.configuration_required: + problems.extend( + _problems( + ("configuration",), + f"{entity} {name!r} requires a 'configuration' object", + "missing_key", + ) + ) + return tuple(problems) + configuration = mapping["configuration"] + if not isinstance(configuration, Mapping): + problems.extend(_problems(("configuration",), f"expected an object, got {configuration!r}")) + return tuple(problems) + config = cast("Mapping[object, object]", configuration) + for key in config: + if not isinstance(key, str) or key not in shape.config_keys: + problems.extend(_problems(("configuration",), f"unexpected key {key!r}", "unknown_key")) + for key in sorted(shape.config_required - {k for k in config if isinstance(k, str)}): + problems.extend( + _problems( + ("configuration", key), + f"configuration for {entity} {name!r} is missing required key {key!r}", + "missing_key", + ) + ) + for key, checker in shape.config_checkers.items(): + if key in config: + problems.extend(checker(config[key], ("configuration", key))) + return tuple(problems) + + +def validate_known_codec_metadata(value: object) -> tuple[ValidationProblem, ...] | None: + """Shape problems for a known-name codec entry, or None if not judged. + + None means the entry has no interpretable name or its name is not a + codec this package defines (extension openness: unknown entities are + not ours to judge). An empty list means `value` is an instance of the + named codec's canonical metadata type. + """ + name = entity_name(value) + if name is None: + return None + shape = _CODEC_SHAPES.get(name) + if shape is None: + return None + return _validate_known_entity(value, name, shape, "codec") + + +def validate_known_chunk_grid_metadata(value: object) -> tuple[ValidationProblem, ...] | None: + """Shape problems for a known-name chunk grid entry, or None if not judged.""" + name = entity_name(value) + if name is None: + return None + shape = _CHUNK_GRID_SHAPES.get(name) + if shape is None: + return None + return _validate_known_entity(value, name, shape, "chunk grid") + + +_ENTITY_SHAPES: Final[Mapping[ExtensionPointField, Mapping[str, _EntityShape]]] = { + DATA_TYPE: _DATA_TYPE_SHAPES, + CODECS: _CODEC_SHAPES, + CHUNK_GRID: _CHUNK_GRID_SHAPES, + CHUNK_KEY_ENCODING: _CHUNK_KEY_ENCODING_SHAPES, +} + + +def validate_known_entity_metadata( + field: ExtensionPointField, value: object +) -> tuple[ValidationProblem, ...] | None: + """Shape problems for an entity known at `field`, or None if not judged.""" + name = entity_name(value) + if name is None: + return None + shape = _ENTITY_SHAPES.get(field, {}).get(canonical_name(field, name)) + if shape is None: + return None + return _validate_known_entity(value, name, shape, field.replace("_", " ").rstrip("s")) + + +def modelled_entities() -> frozenset[tuple[ExtensionPointField, str]]: + """Every `(extension point, name)` with a shape validator.""" + return frozenset((field, name) for field, shapes in _ENTITY_SHAPES.items() for name in shapes) + + +def blocking_problems( + problems: Sequence[ValidationProblem], +) -> tuple[ValidationProblem, ...]: + """The problems that prevent interpreting an entity's fields. + + `unknown_key` problems do not: a member this package does not model + says nothing about the members it does. Rules use this so a single + unrecognized key cannot silently suppress every other judgment about + the same entity — the extra key is still reported, and the geometry + checks still run. + """ + return tuple(problem for problem in problems if problem.kind != "unknown_key") + + +__all__ = [ + "blocking_problems", + "entity_name", + "modelled_entities", + "validate_known_chunk_grid_metadata", + "validate_known_codec_metadata", + "validate_known_entity_metadata", +] diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/codec/__init__.py b/packages/zarr-metadata/src/zarr_metadata/v3/codec/__init__.py index c8a9a150fc..f22a2280f9 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/codec/__init__.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/codec/__init__.py @@ -14,6 +14,9 @@ `codecs` list and in sharding's inner pipelines), import `ZarrV3MetadataFieldJSON` from `zarr_metadata.v3`. +The `kind` submodule sorts the known codec names into the spec's three +pipeline kinds (`array -> array`, `array -> bytes`, `bytes -> bytes`). + See https://zarr-specs.readthedocs.io/en/latest/v3/codecs/index.html """ @@ -22,19 +25,31 @@ from zarr_metadata.v3.codec.cast_value import CastValueCodecMetadata from zarr_metadata.v3.codec.crc32c import Crc32cCodecMetadata from zarr_metadata.v3.codec.gzip import GzipCodecMetadata +from zarr_metadata.v3.codec.kind import ( + ARRAY_ARRAY_CODEC_NAMES, + ARRAY_BYTES_CODEC_NAMES, + BYTES_BYTES_CODEC_NAMES, + CodecKind, + codec_kind_of_name, +) from zarr_metadata.v3.codec.scale_offset import ScaleOffsetCodecMetadata from zarr_metadata.v3.codec.sharding_indexed import ShardingIndexedCodecMetadata from zarr_metadata.v3.codec.transpose import TransposeCodecMetadata from zarr_metadata.v3.codec.zstd import ZstdCodecMetadata __all__ = [ + "ARRAY_ARRAY_CODEC_NAMES", + "ARRAY_BYTES_CODEC_NAMES", + "BYTES_BYTES_CODEC_NAMES", "BloscCodecMetadata", "BytesCodecMetadata", "CastValueCodecMetadata", + "CodecKind", "Crc32cCodecMetadata", "GzipCodecMetadata", "ScaleOffsetCodecMetadata", "ShardingIndexedCodecMetadata", "TransposeCodecMetadata", "ZstdCodecMetadata", + "codec_kind_of_name", ] diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/codec/kind.py b/packages/zarr-metadata/src/zarr_metadata/v3/codec/kind.py new file mode 100644 index 0000000000..a5853c0f97 --- /dev/null +++ b/packages/zarr-metadata/src/zarr_metadata/v3/codec/kind.py @@ -0,0 +1,66 @@ +"""Classify Zarr v3 codecs by pipeline kind. + +The v3 spec sorts codecs into three kinds — `array -> array`, +`array -> bytes`, `bytes -> bytes` — and a pipeline is +`array->array* array->bytes bytes->bytes*`. `codec_kind_of_name` +classifies a known name; unknown names have no kind. + +See https://zarr-specs.readthedocs.io/en/latest/v3/codecs/index.html +""" + +from typing import Final, Literal + +from zarr_metadata.v3.codec.blosc import BLOSC_CODEC_NAME +from zarr_metadata.v3.codec.bytes import BYTES_CODEC_NAME +from zarr_metadata.v3.codec.cast_value import CAST_VALUE_CODEC_NAME +from zarr_metadata.v3.codec.crc32c import CRC32C_CODEC_NAME +from zarr_metadata.v3.codec.gzip import GZIP_CODEC_NAME +from zarr_metadata.v3.codec.scale_offset import SCALE_OFFSET_CODEC_NAME +from zarr_metadata.v3.codec.sharding_indexed import SHARDING_INDEXED_CODEC_NAME +from zarr_metadata.v3.codec.transpose import TRANSPOSE_CODEC_NAME +from zarr_metadata.v3.codec.zstd import ZSTD_CODEC_NAME + +ARRAY_ARRAY_CODEC_NAMES: Final = ( + TRANSPOSE_CODEC_NAME, + CAST_VALUE_CODEC_NAME, + SCALE_OFFSET_CODEC_NAME, +) +"""Tuple of the `name` field values of the known `array -> array` codecs.""" + +ARRAY_BYTES_CODEC_NAMES: Final = (BYTES_CODEC_NAME, SHARDING_INDEXED_CODEC_NAME) +"""Tuple of the `name` field values of the known `array -> bytes` codecs.""" + +BYTES_BYTES_CODEC_NAMES: Final = ( + BLOSC_CODEC_NAME, + CRC32C_CODEC_NAME, + GZIP_CODEC_NAME, + ZSTD_CODEC_NAME, +) +"""Tuple of the `name` field values of the known `bytes -> bytes` codecs.""" + +CodecKind = Literal["array_array", "array_bytes", "bytes_bytes"] +"""The three pipeline positions the v3 spec sorts codecs into.""" + + +def codec_kind_of_name(name: str) -> CodecKind | None: + """The pipeline kind of the codec named `name`, or None if unknown. + + Classifies by name alone, with no judgment of the entry's spelling or + configuration; the rules layer judges those separately. + """ + if name in ARRAY_ARRAY_CODEC_NAMES: + return "array_array" + if name in ARRAY_BYTES_CODEC_NAMES: + return "array_bytes" + if name in BYTES_BYTES_CODEC_NAMES: + return "bytes_bytes" + return None + + +__all__ = [ + "ARRAY_ARRAY_CODEC_NAMES", + "ARRAY_BYTES_CODEC_NAMES", + "BYTES_BYTES_CODEC_NAMES", + "CodecKind", + "codec_kind_of_name", +] diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/raw.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/raw.py index c9c688c9fa..dafd5f0575 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/raw.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/raw.py @@ -13,7 +13,14 @@ RawBytesDataTypeName = NewType("RawBytesDataTypeName", str) """A spec-conformant `r` raw-bytes name (e.g. `"r8"`, `"r16"`).""" -_RAW_BYTES_RE: Final = re.compile(r"^r(\d+)$") +RAW_BYTES_NAME_PATTERN: Final = re.compile(r"^r(\d+)$") +"""The *shape* of a raw-bytes data type name, not its validity. + +Matches every `r` spelling including malformed ones (`r0`, `r12`), so +that a misspelled member of this family is recognized as belonging to it +and reported as a misspelling, rather than passing as an unknown +third-party extension. `raw_bytes_dtype_name` applies the validity rule +on top. Sole owner of this grammar: other modules match through it.""" def raw_bytes_dtype_name(value: str) -> RawBytesDataTypeName: @@ -22,7 +29,7 @@ def raw_bytes_dtype_name(value: str) -> RawBytesDataTypeName: Raises ValueError if `value` is not `r` followed by a positive multiple of 8. """ - match = _RAW_BYTES_RE.fullmatch(value) + match = RAW_BYTES_NAME_PATTERN.fullmatch(value) if match is None: raise ValueError(f"Expected 'r' followed by a positive integer, got {value!r}") bits = int(match.group(1)) @@ -39,6 +46,7 @@ def raw_bytes_dtype_name(value: str) -> RawBytesDataTypeName: __all__ = [ + "RAW_BYTES_NAME_PATTERN", "RawBytesDataTypeName", "RawBytesFillValue", "raw_bytes_dtype_name", diff --git a/packages/zarr-metadata/tests/builder/__init__.py b/packages/zarr-metadata/tests/builder/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/packages/zarr-metadata/tests/builder/test_create.py b/packages/zarr-metadata/tests/builder/test_create.py new file mode 100644 index 0000000000..f40cd56206 --- /dev/null +++ b/packages/zarr-metadata/tests/builder/test_create.py @@ -0,0 +1,254 @@ +"""Tests for the `create_*` document factories.""" + +from __future__ import annotations + +from typing import TYPE_CHECKING + +import pytest +import typing_extensions + +import zarr_metadata +import zarr_metadata.builder +from zarr_metadata.builder._create import ( + create_zarr_v2_array_metadata_json, + create_zarr_v2_consolidated_metadata_json, + create_zarr_v2_group_metadata_json, + create_zarr_v2_zarray_json, + create_zarr_v2_zgroup_json, + create_zarr_v3_array_metadata_json, + create_zarr_v3_consolidated_metadata_json, + create_zarr_v3_group_metadata_json, +) +from zarr_metadata.model import MetadataValidationError + +if TYPE_CHECKING: + from collections.abc import Callable, Mapping + + # Factories differ in the document type they return; these tests only + # ever compare the result as a mapping. + Factory = Callable[..., Mapping[str, object]] + +V3_ARRAY: dict[str, object] = { + "zarr_format": 3, + "node_type": "array", + "shape": (4, 4), + "data_type": "uint8", + "fill_value": 0, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (2, 2)}}, + "chunk_key_encoding": "default", + "codecs": ("bytes",), +} + +V2_ZARRAY: dict[str, object] = { + "zarr_format": 2, + "shape": (4,), + "chunks": (2,), + "dtype": " None: + assert factory(**kwargs) == expected + + +def test_output_shares_no_state_with_arguments() -> None: + grid: dict[str, object] = {"name": "regular", "configuration": {"chunk_shape": (2, 2)}} + document = create_zarr_v3_array_metadata_json(**{**V3_ARRAY, "chunk_grid": grid}) + grid["configuration"]["chunk_shape"] = (9, 9) # caller mutates after the fact + assert document["chunk_grid"]["configuration"]["chunk_shape"] == (2, 2) + + +# -- the package rule, enforced ---------------------------------------------- + +# Public TypedDicts ending in JSON that are field/helper shapes rather than +# documents. Closed by hand, like the naming-grammar vocabulary. +_HELPER_SHAPES = frozenset({"ZarrV3NamedConfigJSON"}) + +# Every public document TypedDict, mapped to its factory. Kept here rather +# than in the package: the mapping exists only so this test can hold the +# two sets equal. +_FACTORIES: dict[str, Factory] = { + "ZarrV2ArrayMetadataJSON": create_zarr_v2_array_metadata_json, + "ZarrV2ConsolidatedMetadataJSON": create_zarr_v2_consolidated_metadata_json, + "ZarrV2GroupMetadataJSON": create_zarr_v2_group_metadata_json, + "ZarrV2ZArrayJSON": create_zarr_v2_zarray_json, + "ZarrV2ZGroupJSON": create_zarr_v2_zgroup_json, + "ZarrV3ArrayMetadataJSON": create_zarr_v3_array_metadata_json, + "ZarrV3ConsolidatedMetadataJSON": create_zarr_v3_consolidated_metadata_json, + "ZarrV3GroupMetadataJSON": create_zarr_v3_group_metadata_json, +} + + +def test_every_document_typeddict_has_a_factory() -> None: + documents = { + name + for name in zarr_metadata.__all__ + if typing_extensions.is_typeddict(getattr(zarr_metadata, name)) + and name.endswith("JSON") + and name not in _HELPER_SHAPES + } + assert documents == set(_FACTORIES) + for factory in _FACTORIES.values(): + assert factory.__name__ in zarr_metadata.builder.__all__ + + +# -- error cases, one test per failure mode ---------------------------------- + + +def test_error_v3_array_semantic_rules_run() -> None: + with pytest.raises(MetadataValidationError, match=r"\[0, 255\]"): + create_zarr_v3_array_metadata_json(**{**V3_ARRAY, "fill_value": 300}) + + +def test_error_v3_array_structural_garbage_from_untyped_caller() -> None: + with pytest.raises(MetadataValidationError) as info: + create_zarr_v3_array_metadata_json(zarr_format="3") # type: ignore[arg-type] + assert {p.loc[0] for p in info.value.problems if p.kind == "missing_key"} >= { + "node_type", + "shape", + } + + +def test_error_v3_array_extension_shadows_standard_key() -> None: + with pytest.raises(MetadataValidationError, match="standard metadata key"): + create_zarr_v3_array_metadata_json(**V3_ARRAY, extensions={"shape": (9,)}) + + +def test_error_v3_group_extension_shadows_standard_key() -> None: + with pytest.raises(MetadataValidationError, match="standard metadata key"): + create_zarr_v3_group_metadata_json( + zarr_format=3, node_type="group", extensions={"attributes": {}} + ) + + +def test_error_v3_consolidated_invalid() -> None: + with pytest.raises(MetadataValidationError): + create_zarr_v3_consolidated_metadata_json( + kind="inline", + must_understand=True, + metadata={}, # type: ignore[typeddict-item] + ) + + +def test_error_v3_consolidated_child_violates_composition_rules() -> None: + child = {**V3_ARRAY, "fill_value": 300} + with pytest.raises(MetadataValidationError) as exc_info: + create_zarr_v3_consolidated_metadata_json( + kind="inline", must_understand=False, metadata={"a": child} + ) + assert [(problem.loc, problem.kind) for problem in exc_info.value.problems] == [ + (("metadata", "a", "fill_value"), "invalid_value") + ] + + +def test_error_v2_array_structural() -> None: + with pytest.raises(MetadataValidationError): + create_zarr_v2_array_metadata_json(**{**V2_ZARRAY, "order": "K"}) # type: ignore[typeddict-item] + + +def test_error_v3_array_malformed_raw_dtype() -> None: + # r names outside the family grammar are misspellings of a known + # family, not unknown extensions, and must not escape judgment. + with pytest.raises(MetadataValidationError, match="positive multiple of 8"): + create_zarr_v3_array_metadata_json(**{**V3_ARRAY, "data_type": "r12", "fill_value": (1,)}) + + +def test_error_v2_zarray_attributes_via_splat() -> None: + # The signature excludes `attributes` statically, but a splatted call + # bypasses that; the runtime backstop must hold the strict shape. + with pytest.raises(MetadataValidationError, match=".zattrs"): + create_zarr_v2_zarray_json(**{**V2_ZARRAY, "attributes": {"unit": "m"}}) + + +def test_error_v2_zgroup_attributes_via_splat() -> None: + splatted: dict[str, object] = {"zarr_format": 2, "attributes": {"unit": "m"}} + with pytest.raises(MetadataValidationError, match=".zattrs"): + create_zarr_v2_zgroup_json(**splatted) + + +def test_error_v2_consolidated_envelope() -> None: + with pytest.raises(MetadataValidationError, match="expected a mapping"): + create_zarr_v2_consolidated_metadata_json( + zarr_consolidated_format=1, + metadata="not a mapping", # type: ignore[typeddict-item] + ) + + +def test_error_v2_consolidated_format_is_not_one() -> None: + with pytest.raises(MetadataValidationError) as exc_info: + create_zarr_v2_consolidated_metadata_json( + zarr_consolidated_format=2, + metadata={}, + ) + assert [(problem.loc, problem.kind) for problem in exc_info.value.problems] == [ + (("zarr_consolidated_format",), "invalid_value") + ] + + +def test_error_v2_consolidated_array_entry_is_invalid() -> None: + with pytest.raises(MetadataValidationError) as exc_info: + create_zarr_v2_consolidated_metadata_json( + zarr_consolidated_format=1, + metadata={"foo/.zarray": {}}, # type: ignore[typeddict-item] + ) + assert any( + problem.loc[:2] == ("metadata", "foo/.zarray") and problem.kind == "missing_key" + for problem in exc_info.value.problems + ) + + +def test_error_v2_consolidated_entry_has_unknown_suffix() -> None: + with pytest.raises(MetadataValidationError, match="metadata file suffix"): + create_zarr_v2_consolidated_metadata_json( + zarr_consolidated_format=1, + metadata={"foo/data": {}}, # type: ignore[typeddict-item] + ) diff --git a/packages/zarr-metadata/tests/model/test_array.py b/packages/zarr-metadata/tests/model/test_array.py index 32c7be053e..f6f3843e94 100644 --- a/packages/zarr-metadata/tests/model/test_array.py +++ b/packages/zarr-metadata/tests/model/test_array.py @@ -1314,16 +1314,23 @@ def test_v2_filters_must_be_codec_sequence_or_none() -> None: assert [(p.loc, p.kind) for p in problems] == [(("filters",), "invalid_type")], bad -def test_v2_shape_and_chunks_must_have_equal_rank() -> None: - """Raw v2 metadata requires one chunk length per array dimension.""" +def test_v2_shape_chunks_rank_agreement_is_not_structural() -> None: + """Whether chunks matches shape's dimensionality is a composition + judgment owned by zarr_metadata.rules; the structural validator and the + model classes deliberately accept the document (it is a lossless, + structurally well-formed representation of what a store may contain).""" doc = dict(ZarrV2ArrayMetadata.create_default(shape=(2, 3)).to_json()) doc["chunks"] = (1,) - assert [(p.loc, p.kind) for p in validate_array_metadata_v2(doc)] == [ + assert validate_array_metadata_v2(doc) == () + parsed = ZarrV2ArrayMetadata.from_key_value({".zarray": json.dumps(doc).encode()}) + assert parsed.chunks == (1,) + + from zarr_metadata import rules + + assert [(p.loc, p.kind) for p in rules.validate_array_metadata_v2(doc)] == [ (("chunks",), "invalid_value") ] - with pytest.raises(MetadataValidationError, match="same number of dimensions"): - ZarrV2ArrayMetadata.from_key_value({".zarray": json.dumps(doc).encode()}) def test_v2_filters_must_be_nonempty_when_present() -> None: @@ -1514,12 +1521,18 @@ def test_shape_rejects_negative_dimensions() -> None: ] -def test_dimension_names_length_must_match_shape() -> None: - """dimension_names must have one entry per dimension of shape.""" +def test_dimension_names_length_is_not_structural() -> None: + """Whether dimension_names matches shape's dimensionality is a + composition judgment owned by zarr_metadata.rules; the structural + validator deliberately accepts the document.""" doc = dict(ZarrV3ArrayMetadata.create_default(shape=(10,)).to_json()) | { "dimension_names": ("x", "y", "z") } - assert [(p.loc, p.kind) for p in validate_array_metadata_v3(doc)] == [ + assert validate_array_metadata_v3(doc) == () + + from zarr_metadata import rules + + assert [(p.loc, p.kind) for p in rules.validate_array_metadata_v3(doc)] == [ (("dimension_names",), "invalid_value") ] diff --git a/packages/zarr-metadata/tests/rules/__init__.py b/packages/zarr-metadata/tests/rules/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/packages/zarr-metadata/tests/rules/test_documents.py b/packages/zarr-metadata/tests/rules/test_documents.py new file mode 100644 index 0000000000..94a49b6171 --- /dev/null +++ b/packages/zarr-metadata/tests/rules/test_documents.py @@ -0,0 +1,127 @@ +"""Tests for the whole-document validation trios in `zarr_metadata.rules`.""" + +from __future__ import annotations + +from typing import TYPE_CHECKING + +import pytest + +from zarr_metadata.model import MetadataValidationError +from zarr_metadata.model import ( + is_array_metadata_v3 as model_is_array_metadata_v3, +) +from zarr_metadata.rules import ( + is_array_metadata_v2, + is_array_metadata_v3, + parse_array_metadata_v2, + parse_array_metadata_v3, + validate_array_metadata_v2, + validate_array_metadata_v3, +) + +if TYPE_CHECKING: + from collections.abc import Callable, Mapping + + from zarr_metadata import ZarrV2ArrayMetadataJSON, ZarrV3ArrayMetadataJSON + from zarr_metadata.model import ValidationProblem + + # The trios are uniform in their inputs (any object) and differ only in + # the document type they hand back, which these tests never depend on. + Validator = Callable[[object], tuple[ValidationProblem, ...]] + Parser = Callable[[object], Mapping[str, object]] + Check = Callable[[object], bool] + +V3_ARRAY: ZarrV3ArrayMetadataJSON = { + "zarr_format": 3, + "node_type": "array", + "shape": (4, 4), + "data_type": "uint8", + "fill_value": 0, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (2, 2)}}, + "chunk_key_encoding": "default", + "codecs": ("bytes",), +} + +V2_ARRAY: ZarrV2ArrayMetadataJSON = { + "zarr_format": 2, + "shape": (4,), + "chunks": (2,), + "dtype": " None: + parsed = parse(doc) + assert validate(parsed) == () + assert check(parsed) is True + shape = doc["shape"] + assert isinstance(shape, (list, tuple)) + assert parsed["shape"] == tuple(shape) + + +def test_error_v3_combined_report() -> None: + # One raise carrying problems from both passes: a structural problem + # (bad node_type) and a composition problem (fill_value vs data_type). + with pytest.raises(MetadataValidationError) as info: + parse_array_metadata_v3({**V3_ARRAY, "node_type": "grid", "fill_value": 300}) + kinds = {(p.loc, p.kind) for p in info.value.problems} + assert (("node_type",), "invalid_value") in kinds + assert (("fill_value",), "invalid_value") in kinds + + +def test_error_v3_dimension_names_reported_once() -> None: + # Regression: this fault used to be reported twice — once by the + # structural validator, once by the composition rule. The check now + # has one owner. + problems = validate_array_metadata_v3({**V3_ARRAY, "dimension_names": ("x",)}) + assert [(p.loc, p.kind) for p in problems] == [(("dimension_names",), "invalid_value")] + + +def test_error_v2_chunks_rank() -> None: + problems = validate_array_metadata_v2({**V2_ARRAY, "chunks": (2, 2)}) + assert [(p.loc, p.kind) for p in problems] == [(("chunks",), "invalid_value")] + + +def test_error_v2_parse_raises() -> None: + with pytest.raises(MetadataValidationError, match="same number of dimensions"): + parse_array_metadata_v2({**V2_ARRAY, "chunks": (2, 2)}) + + +def test_is_functions_are_not_type_guards() -> None: + # A composition-invalid document is still an instance of the TypedDict, + # so the model layer's TypeIs narrows it while the rules layer's plain + # bool judges it. Divergence here is the design, not a bug. + doc = {**V3_ARRAY, "fill_value": 300} + assert model_is_array_metadata_v3(doc) is True + assert is_array_metadata_v3(doc) is False diff --git a/packages/zarr-metadata/tests/rules/test_registry.py b/packages/zarr-metadata/tests/rules/test_registry.py new file mode 100644 index 0000000000..2f748a70c1 --- /dev/null +++ b/packages/zarr-metadata/tests/rules/test_registry.py @@ -0,0 +1,172 @@ +"""Tests for rule registration. + +The registry exists so that a rule cannot be defined without being run. +These tests cover the three ways that could still fail: a rule declaring +dependencies no such document has, an entity whose rules were never +imported, and a document rule set assembled from something other than +the registry. +""" + +from __future__ import annotations + +import pkgutil + +import pytest + +import zarr_metadata.rules._entities as entities +from zarr_metadata.rules import ZARR_V2_ARRAY_RULES, ZARR_V3_ARRAY_RULES, ZARR_V3_GROUP_RULES +from zarr_metadata.rules._registry import ( + dispatched_fields, + document_rule, + entity_rule, + register_document_type, + registered_entities, +) +from zarr_metadata.rules._v3_array import ZARR_V3_ARRAY +from zarr_metadata.v3._extension_points import ( + CHUNK_GRID, + CHUNK_KEY_ENCODING, + CODECS, + DATA_TYPE, + RAW_BYTES_FAMILY, +) +from zarr_metadata.v3._shape import modelled_entities + +# Entities the package models but that carry no composition rules: their +# canonical shape is the whole of what we can say about them. Listed by +# hand, keyed by extension point, so that adding a codec is a deliberate +# choice between "write rules" and "record that there are none", never a +# silent omission. +_RULE_FREE = frozenset( + { + (CODECS, "blosc"), + (CODECS, "cast_value"), + (CODECS, "crc32c"), + (CODECS, "scale_offset"), + (CODECS, "zstd"), + (CHUNK_KEY_ENCODING, "default"), + (CHUNK_KEY_ENCODING, "v2"), + (DATA_TYPE, "bool"), + (DATA_TYPE, "int8"), + (DATA_TYPE, "int16"), + (DATA_TYPE, "int32"), + (DATA_TYPE, "int64"), + (DATA_TYPE, "uint8"), + (DATA_TYPE, "uint16"), + (DATA_TYPE, "uint32"), + (DATA_TYPE, "uint64"), + (DATA_TYPE, "float16"), + (DATA_TYPE, "float32"), + (DATA_TYPE, "float64"), + (DATA_TYPE, "complex64"), + (DATA_TYPE, "complex128"), + (DATA_TYPE, RAW_BYTES_FAMILY), + (DATA_TYPE, "bytes"), + (DATA_TYPE, "string"), + } +) + + +def test_every_shape_modelled_entity_is_accounted_for() -> None: + # Every shape-modelled entity either + # carries rules or is recorded as deliberately rule-free. + assert modelled_entities() == registered_entities() | _RULE_FREE + + +def test_every_shape_modelled_field_has_a_dispatcher() -> None: + # Regression: shapes existed for four extension points but dispatchers + # for only two, so `entity_rule` accepted registrations at `data_type` + # and `chunk_key_encoding` whose rules then silently never ran — the + # exact silent-pass failure the registry exists to prevent. A rule can + # only fire at a field something dispatches. + shape_modelled = {field for field, _ in modelled_entities()} + assert shape_modelled <= dispatched_fields() + + +def test_every_field_with_rules_has_a_dispatcher() -> None: + assert {field for field, _ in registered_entities()} <= dispatched_fields() + + +def test_rule_free_entities_really_have_no_rules() -> None: + # Guards the exclusion list itself: an entity cannot be listed as + # rule-free while quietly carrying rules. + assert registered_entities() & _RULE_FREE == frozenset() + + +def test_rules_are_keyed_by_extension_point_not_name() -> None: + # `bytes` is a core codec and a registered extension data type; a rule + # for one must never fire on the other, so the key carries the field. + assert {(CODECS, "bytes"), (DATA_TYPE, "bytes")} <= modelled_entities() + assert (CODECS, "bytes") in registered_entities() + assert (DATA_TYPE, "bytes") not in registered_entities() + + +def test_every_entity_module_is_imported() -> None: + # The package auto-imports its modules; this asserts the discovery + # actually ran, so a new module cannot sit unimported and inert. + module_names = {info.name for info in pkgutil.iter_modules(entities.__path__)} + assert len(module_names) != 0 + for name in module_names: + assert f"{entities.__name__}.{name}" in __import__("sys").modules + + +@pytest.mark.parametrize("rules", [ZARR_V3_ARRAY_RULES, ZARR_V2_ARRAY_RULES, ZARR_V3_GROUP_RULES]) +def test_rule_sets_are_non_empty(rules: tuple[object, ...]) -> None: + assert len(rules) != 0 + + +def test_error_document_rule_requiring_an_unknown_key() -> None: + # A rule whose dependency is misspelled can never fire, and a rule + # that never fires is indistinguishable from one that always passes. + with pytest.raises(ValueError, match="could never fire"): + + @document_rule(ZARR_V3_ARRAY, frozenset({"shapee"})) + def _misspelled(document: object) -> tuple[()]: # pragma: no cover - never runs + return () + + +def test_error_entity_rule_requiring_an_unknown_key() -> None: + with pytest.raises(ValueError, match="could never fire"): + + @entity_rule(ZARR_V3_ARRAY, CHUNK_GRID, "regular", requires=frozenset({"shapee"})) + def _misspelled(configuration: object, document: object) -> tuple[()]: # pragma: no cover + return () + + +def test_error_entity_rule_for_an_unmodelled_entity() -> None: + # Entity rules read configuration members by name, so a rule for an + # entity with no shape validator could never fire. + with pytest.raises(ValueError, match="no shape validator"): + + @entity_rule(ZARR_V3_ARRAY, CHUNK_GRID, "hilbert") + def _unmodelled(configuration: object, document: object) -> tuple[()]: # pragma: no cover + return () + + +def test_error_entity_rule_for_name_modelled_only_at_another_extension_point() -> None: + # `regular` has a chunk-grid shape, but no codec shape. Name-only lookup + # would accept this registration and later interpret codec metadata using + # the chunk-grid schema. + with pytest.raises(ValueError, match="no shape validator"): + + @entity_rule(ZARR_V3_ARRAY, CODECS, "regular") + def _wrong_extension_point(configuration: object, document: object) -> tuple[()]: + return () + + +def test_error_rule_for_an_unregistered_document_type() -> None: + with pytest.raises(LookupError, match="unknown document type"): + + @document_rule("zarr_v9_array", frozenset()) + def _orphan(document: object) -> tuple[()]: # pragma: no cover - never runs + return () + + +def test_register_document_type_accepts_declared_extension_keys() -> None: + register_document_type("test_doc", frozenset({"a"}), extension_keys=frozenset({"b"})) + + @document_rule("test_doc", frozenset({"a", "b"})) + def _uses_both(document: object) -> tuple[()]: + return () + + assert _uses_both.requires == frozenset({"a", "b"}) diff --git a/packages/zarr-metadata/tests/rules/test_result.py b/packages/zarr-metadata/tests/rules/test_result.py new file mode 100644 index 0000000000..49e859a873 --- /dev/null +++ b/packages/zarr-metadata/tests/rules/test_result.py @@ -0,0 +1,107 @@ +"""Tests for the `check_*` tagged-union entry points.""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, get_args, get_type_hints + +import pytest + +from zarr_metadata.rules import ( + Invalid, + Valid, + ValidationResult, + check_array_metadata_v2, + check_array_metadata_v3, + check_group_metadata_v2, + check_group_metadata_v3, +) +from zarr_metadata.rules._documents import validate_array_metadata_v3 + +if TYPE_CHECKING: + from collections.abc import Callable, Mapping + +V3_ARRAY: Mapping[str, object] = { + "zarr_format": 3, + "node_type": "array", + "shape": (4, 4), + "data_type": "uint8", + "fill_value": 0, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (2, 2)}}, + "chunk_key_encoding": "default", + "codecs": ("bytes",), +} +V2_ARRAY: Mapping[str, object] = { + "zarr_format": 2, + "shape": (4,), + "chunks": (2,), + "dtype": " None: + result = check(doc) + assert isinstance(result, Valid) + assert result.valid is True + assert result.document == doc + + +def test_valid_normalizes_like_parse() -> None: + # A Valid carries the canonical document, not the caller's spelling. + result = check_array_metadata_v3({**V3_ARRAY, "shape": [4, 4], "codecs": ["bytes"]}) + assert isinstance(result, Valid) + assert result.document["shape"] == (4, 4) + assert result.document["codecs"] == ("bytes",) + + +def test_error_invalid_carries_every_problem() -> None: + result = check_array_metadata_v3({**V3_ARRAY, "node_type": "grid", "fill_value": 300}) + assert isinstance(result, Invalid) + assert result.valid is False + assert len(result.problems) != 0 + # the same report validate_* would give, not a summary of it + assert result.problems == validate_array_metadata_v3( + {**V3_ARRAY, "node_type": "grid", "fill_value": 300} + ) + + +def test_error_invalid_cannot_have_an_empty_report() -> None: + with pytest.raises(ValueError, match="at least one"): + Invalid(()) + + +def test_public_result_type_is_runtime_subscriptable() -> None: + assert get_args(ValidationResult[int]) == (Valid[int], Invalid) + + +def test_public_check_annotations_resolve_at_runtime() -> None: + hints = get_type_hints(check_array_metadata_v3) + assert get_args(hints["return"])[1] is Invalid + + +def test_discriminant_narrows_both_ways() -> None: + # The point of the union: `valid` selects which member is readable. + good = check_array_metadata_v3(V3_ARRAY) + if good.valid: + assert good.document["zarr_format"] == 3 + else: # pragma: no cover - the fixture is valid + pytest.fail("expected a Valid result") + + bad = check_array_metadata_v3({**V3_ARRAY, "fill_value": 300}) + if bad.valid: # pragma: no cover - the fixture is invalid + pytest.fail("expected an Invalid result") + else: + assert any(problem.loc == ("fill_value",) for problem in bad.problems) diff --git a/packages/zarr-metadata/tests/rules/test_rule_properties.py b/packages/zarr-metadata/tests/rules/test_rule_properties.py new file mode 100644 index 0000000000..12353cb709 --- /dev/null +++ b/packages/zarr-metadata/tests/rules/test_rule_properties.py @@ -0,0 +1,171 @@ +"""Property tests for composition-rule boundaries and API agreement.""" + +from __future__ import annotations + +from typing import TYPE_CHECKING + +import pytest +from hypothesis import given +from hypothesis import strategies as st + +from zarr_metadata.builder import create_zarr_v3_array_metadata_json +from zarr_metadata.model import MetadataValidationError +from zarr_metadata.rules import ( + validate_array_metadata_v2, + validate_array_metadata_v3, + validate_group_metadata_v2, + validate_group_metadata_v3, +) + +if TYPE_CHECKING: + from collections.abc import Callable, Mapping + + from zarr_metadata.model import ValidationProblem + + +JSON_SCALARS = st.none() | st.booleans() | st.integers() | st.floats(allow_nan=False) | st.text() +JSON_VALUES = st.recursive( + JSON_SCALARS, + lambda children: ( + st.lists(children, max_size=4) | st.dictionaries(st.text(max_size=12), children, max_size=4) + ), + max_leaves=20, +) + + +DOCUMENT_VALIDATORS = ( + validate_array_metadata_v2, + validate_array_metadata_v3, + validate_group_metadata_v2, + validate_group_metadata_v3, +) + + +@pytest.mark.parametrize("validator", DOCUMENT_VALIDATORS, ids=lambda validator: validator.__name__) +@given(JSON_VALUES) +def test_document_validators_are_total_for_arbitrary_json( + validator: Callable[[object], tuple[ValidationProblem, ...]], value: object +) -> None: + """Untrusted JSON always produces a verdict; it never crashes the validator.""" + assert isinstance(validator(value), tuple) + + +@given( + extent=st.integers(min_value=0, max_value=200), + chunk_shapes=st.lists(st.integers(min_value=1, max_value=50), min_size=1, max_size=8), +) +def test_rectilinear_explicit_chunks_may_overflow_extent( + extent: int, chunk_shapes: list[int] +) -> None: + """The final explicit chunk may extend past the array boundary.""" + if sum(chunk_shapes) < extent: + return + document = { + "zarr_format": 3, + "node_type": "array", + "shape": (extent,), + "data_type": "uint8", + "fill_value": 0, + "chunk_grid": { + "name": "rectilinear", + "configuration": {"kind": "inline", "chunk_shapes": (tuple(chunk_shapes),)}, + }, + "chunk_key_encoding": "default", + "codecs": ("bytes",), + } + assert validate_array_metadata_v3(document) == () + + +@given(depth=st.integers(min_value=1, max_value=5), leaf_fill=st.integers(0, 255)) +def test_nested_struct_fill_values_are_checked_recursively(depth: int, leaf_fill: int) -> None: + data_type: object = "uint8" + fill_value: object = leaf_fill + path: list[str] = [] + for level in range(depth): + name = f"level_{level}" + data_type = { + "name": "struct", + "configuration": {"fields": ({"name": name, "data_type": data_type},)}, + } + fill_value = {name: fill_value} + path.insert(0, name) + + document: dict[str, object] = { + "zarr_format": 3, + "node_type": "array", + "shape": (1,), + "data_type": data_type, + "fill_value": fill_value, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (1,)}}, + "chunk_key_encoding": "default", + "codecs": ("bytes",), + } + assert validate_array_metadata_v3(document) == () + + invalid: object = 256 + for name in reversed(path): + invalid = {name: invalid} + document["fill_value"] = invalid + problems = validate_array_metadata_v3(document) + assert any(problem.loc == ("fill_value", *path) for problem in problems) + + +@given( + exponents=st.lists(st.integers(min_value=0, max_value=6), min_size=1, max_size=4, unique=True) +) +def test_nested_sharding_pipelines_accept_divisible_inner_chunks(exponents: list[int]) -> None: + """Every generated nesting level is checked against the level enclosing it.""" + inner_shapes = [2**exponent for exponent in sorted(exponents, reverse=True)] + codecs: tuple[object, ...] = ("bytes",) + for inner_shape in reversed(inner_shapes): + codecs = ( + { + "name": "sharding_indexed", + "configuration": { + "chunk_shape": (inner_shape,), + "codecs": codecs, + "index_codecs": ( + {"name": "bytes", "configuration": {"endian": "little"}}, + "crc32c", + ), + }, + }, + ) + document = { + "zarr_format": 3, + "node_type": "array", + "shape": (64,), + "data_type": "uint8", + "fill_value": 0, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (64,)}}, + "chunk_key_encoding": "default", + "codecs": codecs, + } + assert validate_array_metadata_v3(document) == () + + +@given( + data_type=st.sampled_from(("int8", "uint8", "int16", "uint16", "int32", "uint32")), + fill_value=st.integers(min_value=-(2**40), max_value=2**40), +) +def test_validator_and_factory_agree(data_type: str, fill_value: int) -> None: + document: Mapping[str, object] = { + "zarr_format": 3, + "node_type": "array", + "shape": (4,), + "data_type": data_type, + "fill_value": fill_value, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (2,)}}, + "chunk_key_encoding": "default", + "codecs": ({"name": "bytes", "configuration": {"endian": "little"}},), + } + validator_accepts = validate_array_metadata_v3(document) == () + + try: + create_zarr_v3_array_metadata_json(**document) # type: ignore[arg-type] + except MetadataValidationError: + factory_accepts = False + else: + factory_accepts = True + + assert validator_accepts == factory_accepts diff --git a/packages/zarr-metadata/tests/rules/test_spec_propagation.py b/packages/zarr-metadata/tests/rules/test_spec_propagation.py new file mode 100644 index 0000000000..e61733e8fa --- /dev/null +++ b/packages/zarr-metadata/tests/rules/test_spec_propagation.py @@ -0,0 +1,160 @@ +"""Tests for array-spec propagation through a codec chain. + +The property under test: every codec is judged against the array it +*receives*, which is the document's chunk only for the first codec in +the chain. Anything that transforms the array — a transpose, a cast, a +shard — changes what the next codec sees. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING + +import pytest + +from zarr_metadata.rules import validate_array_metadata_v3 +from zarr_metadata.rules._spec import ( + NOTHING_KNOWN, + ArraySpec, + propagate, + transitions_registered, +) +from zarr_metadata.v3.codec.kind import ARRAY_ARRAY_CODEC_NAMES + +if TYPE_CHECKING: + from collections.abc import Mapping + + +def _doc(codecs: tuple[object, ...], chunk: tuple[int, ...] = (6, 4)) -> Mapping[str, object]: + return { + "zarr_format": 3, + "node_type": "array", + "shape": (12, 8), + "data_type": "uint8", + "fill_value": 0, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": chunk}}, + "chunk_key_encoding": "default", + "codecs": codecs, + } + + +def _transpose(*order: int) -> Mapping[str, object]: + return {"name": "transpose", "configuration": {"order": order}} + + +def _shard( + inner: tuple[int, ...], + codecs: tuple[object, ...] = ("bytes",), + index_codecs: tuple[object, ...] = ({"name": "bytes", "configuration": {"endian": "little"}},), +) -> Mapping[str, object]: + return { + "name": "sharding_indexed", + "configuration": {"chunk_shape": inner, "codecs": codecs, "index_codecs": index_codecs}, + } + + +# (document, expected verdict). The chunk is (6, 4); a transpose (1, 0) in +# front of a shard means the shard receives (4, 6), and the verdict must +# follow the transposed shape, not the grid. +CASES: dict[str, tuple[Mapping[str, object], bool]] = { + "shard-alone-divides": (_doc((_shard((3, 2)),)), True), + "shard-alone-does-not-divide": (_doc((_shard((4, 3)),)), False), + # Regression: before propagation these two verdicts were reversed. + "transpose-then-shard-divides-transposed": (_doc((_transpose(1, 0), _shard((2, 3)))), True), + "transpose-then-shard-does-not-divide-transposed": ( + _doc((_transpose(1, 0), _shard((3, 2)))), + False, + ), + "two-transposes-cancel": (_doc((_transpose(1, 0), _transpose(1, 0), _shard((3, 2)))), True), + # Inside a shard the incoming array is the inner chunk, so a nested + # transpose is judged against the inner chunk's rank, and a nested + # shard against the transposed inner chunk. + "nested-transpose-matches-inner-rank": ( + _doc((_shard((3, 2), codecs=(_transpose(1, 0), "bytes")),)), + True, + ), + "nested-shard-follows-nested-transpose": ( + # inner chunk (3, 2) transposed -> (2, 3); nested shard (2, 1) divides it. + _doc((_shard((3, 2), codecs=(_transpose(1, 0), _shard((2, 1)))),)), + True, + ), + "nested-shard-violates-transposed-inner": ( + # inner chunk (3, 2) transposed -> (2, 3); nested shard (3, 1): 3 does not divide 2. + _doc((_shard((3, 2), codecs=(_transpose(1, 0), _shard((3, 1)))),)), + False, + ), + # An unknown codec might change the shape, so downstream geometry + # declines rather than guessing — an otherwise-invalid shard passes. + "unknown-codec-stops-propagation": (_doc(({"name": "zfpy"}, _shard((4, 3)))), True), + # The shard index is a uint64 array, so a bytes codec inside + # `index_codecs` needs an endianness like any multi-byte encoding. + "index-codecs-bare-bytes-needs-endian": ( + _doc((_shard((3, 2), index_codecs=("bytes", "crc32c")),)), + False, + ), +} + + +@pytest.mark.parametrize(("doc", "valid"), CASES.values(), ids=list(CASES)) +def test_verdict_follows_the_incoming_array(doc: Mapping[str, object], valid: bool) -> None: + problems = validate_array_metadata_v3(doc) + assert (len(problems) == 0) is valid, [str(p) for p in problems] + + +def test_error_locates_the_offending_shard() -> None: + problems = validate_array_metadata_v3(_doc((_transpose(1, 0), _shard((3, 2))))) + assert [(p.loc, p.kind) for p in problems] == [ + (("codecs", 1, "configuration", "chunk_shape", 0), "invalid_value") + ] + + +def test_propagate_yields_incoming_spec_per_codec() -> None: + from zarr_metadata.rules._registry import entity_configuration + from zarr_metadata.v3._extension_points import CODECS + + chain = (_transpose(1, 0), "bytes", "crc32c") + start = ArraySpec((6, 4), "uint8") + seen = list(propagate(chain, start, lambda c: entity_configuration(CODECS, c))) + incoming = [spec for _, _, spec in seen] + assert incoming[0] == ArraySpec((6, 4), "uint8") # transpose receives the chunk + assert incoming[1] == ArraySpec((4, 6), "uint8") # bytes receives the transposed chunk + # past array->bytes: no array, so no shape; the type carries through + assert incoming[2] == ArraySpec(None, "uint8") + + +def test_cast_value_changes_the_downstream_data_type() -> None: + from zarr_metadata.rules._registry import entity_configuration + from zarr_metadata.v3._extension_points import CODECS + + chain = ({"name": "cast_value", "configuration": {"data_type": "float32"}}, "bytes") + start = ArraySpec((6, 4), "uint8") + seen = list(propagate(chain, start, lambda c: entity_configuration(CODECS, c))) + assert seen[1][2] == ArraySpec((6, 4), "float32") + + +def test_unknown_codec_yields_nothing_known() -> None: + from zarr_metadata.rules._registry import entity_configuration + from zarr_metadata.v3._extension_points import CODECS + + start = ArraySpec((6, 4), "uint8") + seen = list( + propagate(({"name": "zfpy"}, "bytes"), start, lambda c: entity_configuration(CODECS, c)) + ) + assert seen[1][2] is NOTHING_KNOWN + + +def test_every_array_array_codec_registers_a_transition() -> None: + # A modelled array->array codec with no transition is treated as + # unknown and stops propagation — safe, but silently weaker than + # intended. Make it a decision, not an omission. + assert set(ARRAY_ARRAY_CODEC_NAMES) <= transitions_registered() | {"scale_offset"} + + +def test_error_transition_for_a_non_array_array_codec() -> None: + from zarr_metadata.rules._spec import spec_transition + + with pytest.raises(ValueError, match="only array->array codecs"): + + @spec_transition("gzip") + def _nope(configuration: object, incoming: ArraySpec) -> ArraySpec: # pragma: no cover + return incoming diff --git a/packages/zarr-metadata/tests/rules/test_v3_array_rules.py b/packages/zarr-metadata/tests/rules/test_v3_array_rules.py new file mode 100644 index 0000000000..e3403726d5 --- /dev/null +++ b/packages/zarr-metadata/tests/rules/test_v3_array_rules.py @@ -0,0 +1,553 @@ +"""Tests for the v3 array and group composition rules added with the +rules-layer promotion: chunk grid values/geometry, transpose orders, +sharding pipelines/geometry, and consolidated-entry recursion.""" + +from __future__ import annotations + +from typing import TYPE_CHECKING + +import pytest + +from zarr_metadata.model import MetadataValidationError +from zarr_metadata.rules import validate_array_metadata_v3, validate_group_metadata_v3 + +if TYPE_CHECKING: + from collections.abc import Mapping + + from zarr_metadata import ZarrV3ArrayMetadataJSON + +BASE: ZarrV3ArrayMetadataJSON = { + "zarr_format": 3, + "node_type": "array", + "shape": (4, 4), + "data_type": "uint8", + "fill_value": 0, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (2, 2)}}, + "chunk_key_encoding": "default", + "codecs": ("bytes",), +} + + +# The shard index is a uint64 array, so its bytes codec needs an endianness. +_INDEX_BYTES: Mapping[str, object] = {"name": "bytes", "configuration": {"endian": "little"}} + + +def _shard(**overrides: object) -> Mapping[str, object]: + """A sharding codec entry; overrides may be deliberately malformed.""" + configuration: dict[str, object] = { + "chunk_shape": (2, 2), + "codecs": ("bytes",), + "index_codecs": (_INDEX_BYTES, "crc32c"), + } + configuration.update(overrides) + return {"name": "sharding_indexed", "configuration": configuration} + + +# Documents that must be fully valid: the rules judge geometry and values +# without rejecting legitimate spellings of the same constructs. +VALID_CASES: dict[str, Mapping[str, object]] = { + "regular": BASE, + "rectilinear-explicit-sums": { + **BASE, + "chunk_grid": { + "name": "rectilinear", + "configuration": {"kind": "inline", "chunk_shapes": ((2, 2), (1, 3))}, + }, + }, + "rectilinear-explicit-overflow": { + **BASE, + "shape": (6,), + "chunk_grid": { + "name": "rectilinear", + "configuration": {"kind": "inline", "chunk_shapes": ((4, 4, 4),)}, + }, + }, + "rectilinear-rle-and-uniform": { + **BASE, + "chunk_grid": { + "name": "rectilinear", + "configuration": {"kind": "inline", "chunk_shapes": (((2, 2),), 4)}, + }, + }, + "transpose": { + **BASE, + "codecs": ({"name": "transpose", "configuration": {"order": (1, 0)}}, "bytes"), + }, + "sharding": {**BASE, "codecs": (_shard(),)}, + "nested-sharding": { + **BASE, + "codecs": ( + _shard( + codecs=( + { + "name": "sharding_indexed", + "configuration": { + "chunk_shape": (1, 2), + "codecs": ("bytes",), + "index_codecs": (_INDEX_BYTES,), + }, + }, + ) + ), + ), + }, + "nested-struct": { + **BASE, + "data_type": { + "name": "struct", + "configuration": { + "fields": ( + {"name": "id", "data_type": "uint8"}, + { + "name": "point", + "data_type": { + "name": "struct", + "configuration": {"fields": ({"name": "x", "data_type": "int16"},)}, + }, + }, + ) + }, + }, + "fill_value": {"id": 1, "point": {"x": -2}}, + "codecs": ({"name": "bytes", "configuration": {"endian": "little"}},), + }, + "unknown-grid-passes": { + **BASE, + "chunk_grid": {"name": "hilbert", "configuration": {"level": 3}}, + }, + "unknown-codec-inconclusive": {**BASE, "codecs": ({"name": "zfpy"}, "bytes")}, +} + + +@pytest.mark.parametrize("doc", VALID_CASES.values(), ids=list(VALID_CASES)) +def test_valid_documents(doc: Mapping[str, object]) -> None: + assert validate_array_metadata_v3(doc) == () + + +def _sole_problem(doc: Mapping[str, object]) -> tuple[tuple[str | int, ...], str]: + problems = validate_array_metadata_v3(doc) + assert len(problems) == 1, [p.message for p in problems] + return problems[0].loc, problems[0].message + + +def test_error_regular_chunk_extent_zero() -> None: + loc, message = _sole_problem( + {**BASE, "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (0, 2)}}} + ) + assert loc == ("chunk_grid", "configuration", "chunk_shape", 0) + assert "positive chunk extent" in message + + +def test_error_rectilinear_rank_mismatch() -> None: + loc, message = _sole_problem( + { + **BASE, + "chunk_grid": { + "name": "rectilinear", + "configuration": {"kind": "inline", "chunk_shapes": ((2, 2),)}, + }, + } + ) + assert loc == ("chunk_grid", "configuration", "chunk_shapes") + assert "2 dimensions" in message + + +def test_error_rectilinear_sum_mismatch() -> None: + loc, message = _sole_problem( + { + **BASE, + "chunk_grid": { + "name": "rectilinear", + "configuration": {"kind": "inline", "chunk_shapes": ((3,), (2, 2))}, + }, + } + ) + assert loc == ("chunk_grid", "configuration", "chunk_shapes", 0) + assert "sum to 3" in message + + +def test_error_rectilinear_nonpositive_rle() -> None: + problems = validate_array_metadata_v3( + { + **BASE, + "chunk_grid": { + "name": "rectilinear", + "configuration": {"kind": "inline", "chunk_shapes": (((0, 4),), 4)}, + }, + } + ) + assert any("positive [size, count] pair" in p.message for p in problems) + + +def test_error_transpose_not_a_permutation() -> None: + loc, message = _sole_problem( + {**BASE, "codecs": ({"name": "transpose", "configuration": {"order": (5, 5)}}, "bytes")} + ) + assert loc == ("codecs", 0, "configuration", "order") + assert "permutation" in message + + +def test_error_transpose_rank_mismatch() -> None: + loc, message = _sole_problem( + {**BASE, "codecs": ({"name": "transpose", "configuration": {"order": (2, 0, 1)}}, "bytes")} + ) + assert loc == ("codecs", 0, "configuration", "order") + assert "incoming array has 2 dimensions" in message + + +def test_error_sharding_inner_pipeline_order() -> None: + loc, _ = _sole_problem({**BASE, "codecs": (_shard(codecs=("crc32c", "bytes")),)}) + assert loc == ("codecs", 0, "configuration", "codecs", 1) + + +def test_error_sharding_inner_no_array_bytes() -> None: + loc, message = _sole_problem({**BASE, "codecs": (_shard(codecs=("crc32c",)),)}) + assert loc == ("codecs", 0, "configuration", "codecs") + assert "no array->bytes codec" in message + + +def test_error_sharding_index_codecs_no_array_bytes() -> None: + loc, message = _sole_problem({**BASE, "codecs": (_shard(index_codecs=("crc32c",)),)}) + assert loc == ("codecs", 0, "configuration", "index_codecs") + assert "no array->bytes codec" in message + + +def test_error_sharding_index_codecs_are_variable_sized() -> None: + loc, message = _sole_problem( + { + **BASE, + "codecs": ( + _shard( + index_codecs=( + _INDEX_BYTES, + {"name": "gzip", "configuration": {"level": 1}}, + ) + ), + ), + } + ) + assert loc == ("codecs", 0, "configuration", "index_codecs", 1) + assert "fixed-size" in message + + +def test_error_sharding_rank_mismatch() -> None: + loc, message = _sole_problem({**BASE, "codecs": (_shard(chunk_shape=(2,)),)}) + assert loc == ("codecs", 0, "configuration", "chunk_shape") + assert "incoming array has 2 dimensions" in message + + +def test_error_sharding_not_divisible() -> None: + loc, message = _sole_problem( + { + **BASE, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (4, 4)}}, + "codecs": (_shard(chunk_shape=(3, 2)),), + } + ) + assert loc == ("codecs", 0, "configuration", "chunk_shape", 0) + assert "does not evenly divide" in message + + +def test_error_nested_sharding_not_divisible() -> None: + inner: Mapping[str, object] = { + "name": "sharding_indexed", + "configuration": { + "chunk_shape": (2, 3), + "codecs": ("bytes",), + "index_codecs": (_INDEX_BYTES,), + }, + } + loc, message = _sole_problem( + { + **BASE, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (4, 4)}}, + "codecs": (_shard(codecs=(inner,)),), + } + ) + assert loc == ("codecs", 0, "configuration", "codecs", 0, "configuration", "chunk_shape", 1) + assert "does not evenly divide" in message + + +def test_error_sharding_inner_chunk_extent_zero() -> None: + problems = validate_array_metadata_v3({**BASE, "codecs": (_shard(chunk_shape=(0, 2)),)}) + assert any( + p.loc == ("codecs", 0, "configuration", "chunk_shape", 0) + and "positive chunk extent" in p.message + for p in problems + ) + + +def test_error_bytes_requires_endian_for_multibyte_data() -> None: + loc, message = _sole_problem({**BASE, "data_type": "int32", "codecs": ("bytes",)}) + assert loc == ("codecs", 0, "configuration", "endian") + assert "required" in message + + +def test_error_bytes_rejects_variable_length_data_type() -> None: + loc, message = _sole_problem( + {**BASE, "data_type": "string", "fill_value": "", "codecs": ("bytes",)} + ) + assert loc == ("codecs", 0, "configuration") + assert "not compatible" in message + + +def test_error_struct_fields_are_empty() -> None: + doc = { + **BASE, + "data_type": {"name": "struct", "configuration": {"fields": ()}}, + "fill_value": {}, + } + loc, message = _sole_problem(doc) + assert loc == ("data_type", "configuration", "fields") + assert "at least one" in message + + +def test_error_struct_field_is_variable_length() -> None: + doc = { + **BASE, + "data_type": { + "name": "struct", + "configuration": {"fields": ({"name": "label", "data_type": "string"},)}, + }, + "fill_value": {"label": ""}, + } + problems = validate_array_metadata_v3(doc) + assert any( + problem.loc == ("data_type", "configuration", "fields", 0, "data_type") + and "fixed-size" in problem.message + for problem in problems + ) + + +def test_error_struct_fill_is_missing_field() -> None: + doc = { + **BASE, + "data_type": { + "name": "struct", + "configuration": {"fields": ({"name": "x", "data_type": "uint8"},)}, + }, + "fill_value": {}, + } + loc, message = _sole_problem(doc) + assert loc == ("fill_value", "x") + assert "missing" in message + + +def test_error_struct_fill_field_is_invalid() -> None: + doc = { + **BASE, + "data_type": { + "name": "struct", + "configuration": {"fields": ({"name": "x", "data_type": "uint8"},)}, + }, + "fill_value": {"x": 300}, + } + loc, message = _sole_problem(doc) + assert loc == ("fill_value", "x") + assert "[0, 255]" in message + + +def test_error_gzip_level_is_out_of_range() -> None: + doc = { + **BASE, + "codecs": ("bytes", {"name": "gzip", "configuration": {"level": 99}}), + } + loc, message = _sole_problem(doc) + assert loc == ("codecs", 1, "configuration", "level") + assert "[0, 9]" in message + + +@pytest.mark.parametrize("data_type_name", ["numpy.datetime64", "numpy.timedelta64"]) +def test_error_numpy_time_scale_factor_is_out_of_range(data_type_name: str) -> None: + doc = { + **BASE, + "data_type": { + "name": data_type_name, + "configuration": {"unit": "ns", "scale_factor": 0}, + }, + "codecs": ({"name": "bytes", "configuration": {"endian": "little"}},), + } + loc, message = _sole_problem(doc) + assert loc == ("data_type", "configuration", "scale_factor") + assert "[1, 2147483647]" in message + + +@pytest.mark.parametrize("data_type_name", ["numpy.datetime64", "numpy.timedelta64"]) +def test_error_numpy_time_fill_is_out_of_range(data_type_name: str) -> None: + doc = { + **BASE, + "data_type": { + "name": data_type_name, + "configuration": {"unit": "ns", "scale_factor": 1}, + }, + "fill_value": 2**80, + "codecs": ({"name": "bytes", "configuration": {"endian": "little"}},), + } + loc, message = _sole_problem(doc) + assert loc == ("fill_value",) + assert "64-bit" in message + + +def test_error_consolidated_child_violates_array_rules() -> None: + doc: Mapping[str, object] = { + "zarr_format": 3, + "node_type": "group", + "consolidated_metadata": { + "kind": "inline", + "must_understand": False, + "metadata": {"a": {**BASE, "fill_value": 300}}, + }, + } + problems = validate_group_metadata_v3(doc) + assert [(p.loc, p.kind) for p in problems] == [ + (("consolidated_metadata", "metadata", "a", "fill_value"), "invalid_value") + ] + + +def test_error_consolidated_nested_group_recursion() -> None: + child_group: Mapping[str, object] = { + "zarr_format": 3, + "node_type": "group", + "consolidated_metadata": { + "kind": "inline", + "must_understand": False, + "metadata": {"b": {**BASE, "fill_value": 300}}, + }, + } + doc: Mapping[str, object] = { + "zarr_format": 3, + "node_type": "group", + "consolidated_metadata": { + "kind": "inline", + "must_understand": False, + "metadata": {"g": child_group}, + }, + } + problems = validate_group_metadata_v3(doc) + assert [p.loc for p in problems] == [ + ( + "consolidated_metadata", + "metadata", + "g", + "consolidated_metadata", + "metadata", + "b", + "fill_value", + ) + ] + + +def test_error_group_parse_raises() -> None: + from zarr_metadata.rules import parse_group_metadata_v3 + + with pytest.raises(MetadataValidationError, match="fill_value invalid"): + parse_group_metadata_v3( + { + "zarr_format": 3, + "node_type": "group", + "consolidated_metadata": { + "kind": "inline", + "must_understand": False, + "metadata": {"a": {**BASE, "fill_value": 300}}, + }, + } + ) + + +# -- unknown configuration members -------------------------------------------- +# +# The v3 spec does not say whether an extension's `configuration` is closed +# (zarr-developers/zarr-specs#270, open since 2023). This package takes the +# strict reading, matching most registered extension schemas and most other +# implementations — but reports it as its own `unknown_key` kind, and never +# lets it mask a real finding about the same entity. + + +def test_unknown_configuration_member_has_its_own_kind() -> None: + doc = { + **BASE, + "codecs": ({"name": "bytes", "configuration": {"endian": "little", "hint": 1}},), + } + problems = validate_array_metadata_v3(doc) + assert [(p.loc, p.kind) for p in problems] == [(("codecs", 0, "configuration"), "unknown_key")] + + +def test_error_known_data_type_has_invalid_configuration() -> None: + doc = { + **BASE, + "data_type": { + "name": "numpy.datetime64", + "configuration": {"unit": "banana", "scale_factor": 1}, + }, + } + problems = validate_array_metadata_v3(doc) + assert [(p.loc, p.kind) for p in problems] == [ + (("data_type", "configuration", "unit"), "invalid_value") + ] + + +def test_error_known_chunk_key_encoding_has_invalid_configuration() -> None: + doc = { + **BASE, + "chunk_key_encoding": {"name": "default", "configuration": {"separator": "!"}}, + } + problems = validate_array_metadata_v3(doc) + assert [(p.loc, p.kind) for p in problems] == [ + (("chunk_key_encoding", "configuration", "separator"), "invalid_value") + ] + + +def test_unknown_member_does_not_mask_a_codec_rule() -> None: + # Regression: an unrecognized member used to make the whole entity + # uninterpretable, silently suppressing every other rule about it — so a + # cosmetic extra key hid a genuine permutation error. + doc = { + **BASE, + "codecs": ({"name": "transpose", "configuration": {"order": (5, 5), "hint": 1}}, "bytes"), + } + kinds = {(p.loc, p.kind) for p in validate_array_metadata_v3(doc)} + assert (("codecs", 0, "configuration"), "unknown_key") in kinds + assert (("codecs", 0, "configuration", "order"), "invalid_value") in kinds + + +def test_unknown_member_does_not_mask_a_chunk_grid_rule() -> None: + doc = { + **BASE, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (2,), "hint": 1}}, + } + kinds = {(p.loc, p.kind) for p in validate_array_metadata_v3(doc)} + assert (("chunk_grid", "configuration"), "unknown_key") in kinds + assert (("chunk_grid", "configuration", "chunk_shape"), "invalid_value") in kinds + + +def test_unknown_member_survives_a_round_trip() -> None: + # Whatever the strict validator says, the package must never silently + # drop a member it does not model: a writer that knows more than we do + # must get its bytes back. (zarr-python's own chunk-grid path is lossy + # here; this asserts we are not.) + import json + + from zarr_metadata.model import ZarrV3ArrayMetadata + + raw = { + **BASE, + "shape": [4, 4], + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": [2, 2]}}, + "codecs": [ + { + "name": "blosc", + "configuration": { + "cname": "zstd", + "clevel": 5, + "shuffle": "shuffle", + "blocksize": 0, + "numThreads": 4, + }, + }, + "bytes", + ], + } + model = ZarrV3ArrayMetadata.from_json(json.loads(json.dumps(raw))) + emitted = model.to_json() + codec = emitted["codecs"][0] + assert codec["configuration"]["numThreads"] == 4 diff --git a/packages/zarr-metadata/tests/test_public_api.py b/packages/zarr-metadata/tests/test_public_api.py index 6613aa394b..89be982958 100644 --- a/packages/zarr-metadata/tests/test_public_api.py +++ b/packages/zarr-metadata/tests/test_public_api.py @@ -281,23 +281,29 @@ def test_all_is_grouped_and_unique() -> None: "BloscShuffle", "CastOutOfRangeMode", "CastRoundingMode", + "CodecKind", "Endianness", "HexFloat16", "HexFloat32", "HexFloat64", "JSONValue", + "Invalid", "MetadataValidationError", "NumpyDatetime64", "NumpyTimeUnit", "NumpyTimedelta64", "ProblemKind", "RectilinearDimSpec", + "Rule", + "RuleCheck", "ScalarMap", "ScalarMapEntry", "ShardingIndexLocation", "Struct", "StructField", + "Valid", "ValidationProblem", + "ValidationResult", } ) @@ -409,7 +415,14 @@ def _literal_backed_constants() -> list[tuple[str, str, str]]: for const_name, value in vars(module).items(): if const_name.startswith("_") or not const_name.isupper(): continue - members = frozenset(value) if isinstance(value, tuple) else frozenset({value}) + if isinstance(value, tuple): + members = frozenset(value) + elif isinstance(value, str): + members = frozenset({value}) + else: + # Unhashable non-value constants (rule sets, factory + # registries) back no Literal type and carry no signal. + continue if not all(isinstance(m, str) for m in members): continue matches = [t for t, args in literals.items() if args == members] @@ -441,7 +454,14 @@ def _value_tied_constants() -> set[str]: for const_name, value in vars(module).items(): if const_name.startswith("_") or not const_name.isupper(): continue - members = frozenset(value) if isinstance(value, tuple) else frozenset({value}) + if isinstance(value, tuple): + members = frozenset(value) + elif isinstance(value, str): + members = frozenset({value}) + else: + # Unhashable non-value constants (rule sets, factory + # registries) back no Literal type and carry no signal. + continue if not all(isinstance(m, str) for m in members): continue if sum(1 for args in literals if args == members) > 1: diff --git a/packages/zarr-metadata/tests/test_registry_drift.py b/packages/zarr-metadata/tests/test_registry_drift.py new file mode 100644 index 0000000000..f21851b990 --- /dev/null +++ b/packages/zarr-metadata/tests/test_registry_drift.py @@ -0,0 +1,85 @@ +"""Drift tests tying the hand-written judgment registries to the package's +type modules: adding a codec, chunk grid, or data type module without +registering it in the corresponding judgment surface must fail a test +rather than silently weaken validation (an unregistered codec, for +example, would suppress the exactly-one-array->bytes check for every +pipeline containing it).""" + +from __future__ import annotations + +import importlib +import pkgutil + +import zarr_metadata.v3.chunk_grid +import zarr_metadata.v3.chunk_key_encoding +import zarr_metadata.v3.codec +import zarr_metadata.v3.data_type +from zarr_metadata.rules._v3_array import ( + _check_fill_for_dtype, # pyright: ignore[reportPrivateUsage] +) +from zarr_metadata.v3._extension_points import RAW_BYTES_FAMILY +from zarr_metadata.v3._shape import ( # pyright: ignore[reportPrivateUsage] + _CHUNK_GRID_SHAPES, + _CHUNK_KEY_ENCODING_SHAPES, + _CODEC_SHAPES, + _DATA_TYPE_SHAPES, +) +from zarr_metadata.v3.codec.kind import codec_kind_of_name + + +def _module_constants(package: object, suffix: str) -> set[str]: + """Values of `*` constants across a package's public modules.""" + names: set[str] = set() + for info in pkgutil.iter_modules(package.__path__): # type: ignore[attr-defined] + if info.name.startswith("_"): + continue + module = importlib.import_module(f"{package.__name__}.{info.name}") # type: ignore[attr-defined] + names.update( + value + for attribute, value in vars(module).items() + if attribute.endswith(suffix) and isinstance(value, str) + ) + return names + + +def test_every_codec_module_is_kind_classified() -> None: + codec_names = _module_constants(zarr_metadata.v3.codec, "_CODEC_NAME") + assert codec_names, "constant scan found nothing — the naming convention moved?" + unclassified = {name for name in codec_names if codec_kind_of_name(name) is None} + assert not unclassified + + +def test_every_codec_module_has_a_shape_validator() -> None: + codec_names = _module_constants(zarr_metadata.v3.codec, "_CODEC_NAME") + assert codec_names == set(_CODEC_SHAPES) + + +def test_every_chunk_grid_module_has_a_shape_validator() -> None: + grid_names = _module_constants(zarr_metadata.v3.chunk_grid, "_CHUNK_GRID_NAME") + assert grid_names, "constant scan found nothing — the naming convention moved?" + assert grid_names == set(_CHUNK_GRID_SHAPES) + + +def test_every_chunk_key_encoding_module_has_a_shape_validator() -> None: + names = _module_constants(zarr_metadata.v3.chunk_key_encoding, "_CHUNK_KEY_ENCODING_NAME") + assert names, "constant scan found nothing — the naming convention moved?" + assert names == set(_CHUNK_KEY_ENCODING_SHAPES) + + +def test_every_data_type_module_has_a_shape_validator() -> None: + names = _module_constants(zarr_metadata.v3.data_type, "_DATA_TYPE_NAME") + assert names, "constant scan found nothing — the naming convention moved?" + assert names | {RAW_BYTES_FAMILY} == set(_DATA_TYPE_SHAPES) + + +def test_every_data_type_has_a_fill_value_branch() -> None: + # object() is a valid fill value for no data type this package + # defines, so a known name must produce a complaint; only genuinely + # unknown names may decline (extension openness). The parameterized + # r family has no name constant and is represented by "r8". + dtype_names = _module_constants(zarr_metadata.v3.data_type, "_DATA_TYPE_NAME") + assert dtype_names, "constant scan found nothing — the naming convention moved?" + unjudged = { + name for name in {*dtype_names, "r8"} if _check_fill_for_dtype(name, object()) is None + } + assert not unjudged diff --git a/packages/zarr-metadata/tests/v3/codec/test_kind.py b/packages/zarr-metadata/tests/v3/codec/test_kind.py new file mode 100644 index 0000000000..996d469035 --- /dev/null +++ b/packages/zarr-metadata/tests/v3/codec/test_kind.py @@ -0,0 +1,26 @@ +"""Tests for codec kind classification.""" + +from __future__ import annotations + +import pytest + +from zarr_metadata.v3.codec.kind import codec_kind_of_name + +# (codec name, expected kind). Classification is by name alone. +CASES: dict[str, str | None] = { + "transpose": "array_array", + "cast_value": "array_array", + "scale_offset": "array_array", + "bytes": "array_bytes", + "sharding_indexed": "array_bytes", + "blosc": "bytes_bytes", + "crc32c": "bytes_bytes", + "gzip": "bytes_bytes", + "zstd": "bytes_bytes", + "lightspeed": None, +} + + +@pytest.mark.parametrize(("name", "kind"), CASES.items(), ids=list(CASES)) +def test_kind_of_name(name: str, kind: str | None) -> None: + assert codec_kind_of_name(name) == kind diff --git a/packages/zarr-metadata/tests/v3/test_extension_points.py b/packages/zarr-metadata/tests/v3/test_extension_points.py new file mode 100644 index 0000000000..7a539fe28d --- /dev/null +++ b/packages/zarr-metadata/tests/v3/test_extension_points.py @@ -0,0 +1,76 @@ +"""Tests for extension-point name canonicalization.""" + +from __future__ import annotations + +import pytest + +from zarr_metadata.rules import validate_array_metadata_v3 +from zarr_metadata.v3._extension_points import ( + CODECS, + DATA_TYPE, + RAW_BYTES_FAMILY, + canonical_name, +) + +# (field, name, expected canonical key) — identity everywhere except the +# parameterized raw-bytes family. +CANONICAL_CASES: dict[str, tuple[str, str, str]] = { + "plain-dtype": (DATA_TYPE, "uint8", "uint8"), + "dotted-dtype": (DATA_TYPE, "numpy.datetime64", "numpy.datetime64"), + "raw-8": (DATA_TYPE, "r8", RAW_BYTES_FAMILY), + "raw-24": (DATA_TYPE, "r24", RAW_BYTES_FAMILY), + # Malformed members canonicalize into the family too: a misspelling of + # something we model must be reported as such, not pass as an unknown + # third-party extension. + "raw-not-multiple-of-8": (DATA_TYPE, "r12", RAW_BYTES_FAMILY), + "raw-zero": (DATA_TYPE, "r0", RAW_BYTES_FAMILY), + # Canonicalization is field-aware: the r family is a data type. + "raw-shaped-codec-name": (CODECS, "r8", "r8"), + "codec": (CODECS, "blosc", "blosc"), + "unknown": (CODECS, "zfpy", "zfpy"), +} + + +@pytest.mark.parametrize( + ("field", "name", "expected"), CANONICAL_CASES.values(), ids=list(CANONICAL_CASES) +) +def test_canonical_name(field: str, name: str, expected: str) -> None: + assert canonical_name(field, name) == expected # type: ignore[arg-type] + + +def test_squatted_names_are_judged_against_the_definition_they_squat() -> None: + # Zarr identifiers are registry-allocated. A private codec named + # `bytes` has left the compatibility contract, and saying so is the + # correct answer rather than a limitation, so nothing here defends + # against collisions. + document = { + "zarr_format": 3, + "node_type": "array", + "shape": (4, 4), + "data_type": "uint8", + "fill_value": 0, + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (2, 2)}}, + "chunk_key_encoding": "default", + "codecs": ({"name": "bytes", "configuration": {"width": 7}},), + } + problems = validate_array_metadata_v3(document) + assert [(p.loc, p.kind) for p in problems] == [(("codecs", 0, "configuration"), "unknown_key")] + + +def test_forging_the_family_sentinel_cannot_change_a_verdict() -> None: + # A literal "r" data type mislabels nothing: the rules layer matches + # the family through the name pattern, not through the table key, so + # no validation verdict depends on the sentinel being unforgeable. + assert canonical_name(DATA_TYPE, RAW_BYTES_FAMILY) == RAW_BYTES_FAMILY + document = { + "zarr_format": 3, + "node_type": "array", + "shape": (4, 4), + "data_type": RAW_BYTES_FAMILY, + "fill_value": (1,), + "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": (2, 2)}}, + "chunk_key_encoding": "default", + "codecs": ("bytes",), + } + # Unjudged as an unknown data type, exactly as any unmodelled name is. + assert validate_array_metadata_v3(document) == () diff --git a/packages/zarr-metadata/tests/v3/test_shape_properties.py b/packages/zarr-metadata/tests/v3/test_shape_properties.py new file mode 100644 index 0000000000..1bcf8a3bf2 --- /dev/null +++ b/packages/zarr-metadata/tests/v3/test_shape_properties.py @@ -0,0 +1,51 @@ +"""Generative invariants for raw-bytes name canonicalization. + +Scoped to canonicalization deliberately. Two earlier tests here asserted +that a shape verdict exists exactly when `(field, canonical_name(...))` +is in `modelled_entities()` — but both sides were computed from +`_ENTITY_SHAPES` through the same call, so they restated the lookup +rather than testing it, and could not fail. Worse, they could not catch +the bug class they named (a lookup passing the wrong field), because both +sides used the same field. `tests/rules/test_registry.py` covers that +with real assertions. + +Canonicalization is a genuine fit for generative testing: the family is +unbounded, so an example-based test can only sample it. +""" + +from __future__ import annotations + +from hypothesis import given +from hypothesis import strategies as st + +from zarr_metadata.v3._extension_points import ( + CHUNK_GRID, + CODECS, + DATA_TYPE, + RAW_BYTES_FAMILY, + canonical_name, +) + + +@given(width=st.integers(min_value=0, max_value=2**32)) +def test_every_numeric_r_spelling_folds_to_one_key(width: int) -> None: + # Including malformed widths (0, 12, anything not a multiple of 8): + # canonicalization is by grammar shape, not validity, so a misspelled + # member of a family we model is reported as a misspelling rather than + # passing as an unknown third-party extension. + assert canonical_name(DATA_TYPE, f"r{width}") == RAW_BYTES_FAMILY + + +@given(width=st.integers(min_value=0, max_value=2**32), field=st.sampled_from([CODECS, CHUNK_GRID])) +def test_r_shaped_names_are_identity_outside_data_types(width: int, field: str) -> None: + # The family belongs to `data_type`; a codec that happens to be named + # `r8` must not be folded into it. + name = f"r{width}" + assert canonical_name(field, name) == name # type: ignore[arg-type] + + +@given( + name=st.text(min_size=1).filter(lambda s: not (s.startswith("r") and s[1:].isdigit())), +) +def test_non_family_names_are_identity(name: str) -> None: + assert canonical_name(DATA_TYPE, name) == name