feat(zarr-metadata): composition rules layer, shape-exact entity validators, create_* factories - #318
Open
d-v-b wants to merge 2 commits into
Open
feat(zarr-metadata): composition rules layer, shape-exact entity validators, create_* factories#318d-v-b wants to merge 2 commits into
d-v-b wants to merge 2 commits into
Conversation
…dators, create_* factories Adds `zarr_metadata.rules`: cross-field judgments over whole documents (fill value vs data type, codec pipeline ordering, chunk-grid geometry, sharding and transpose against the array each codec actually receives, struct field rules, dimension-name counts), registered where they are defined and dispatched per entity; `validate_*` / `is_*` / `parse_*` / `check_*` front doors for readers; `v3._shape` validators derived from the entity TypedDicts; and one `create_*` factory per document TypedDict. The two rank checks (v3 `dimension_names` vs `shape`, v2 `chunks` vs `shape`) move from the structural validator into the rules layer. The pydantic field types now run the rules layer before normalizing, so they are strictly stronger than before rather than weaker. Relative to #296 this drops the incremental builder, the extension-point provenance table, `TypeIs` codec guards, fill-value propagation through codec chains (no rule read it), registry introspection helpers, and a duplicated v2 consolidated envelope check; renames the `.zarray`/`.zgroup` factories to `zarray`/`zgroup`; and requires an endianness on the `bytes` codec inside `index_codecs`. Split from #296 (part 2 of 3). Assisted-by: ClaudeCode:claude-fable-5-1
Assisted-by: ClaudeCode:claude-fable-5-1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
Part 2 of 3, split from #296. Stacked on #317; retarget to
mainonce that merges.What this adds
zarr_metadata.rules: composition judgments over whole documents. Fill value vs data type for every dtype family; codec pipeline ordering (AA* AB BB*, exactly one AB, inconclusive with unknown codecs); known-name configuration shapes;dimension_namescount; chunk-grid values and geometry (regular rank; rectilinear rank and per-dimension sums incl. RLE pairs); transpose orders; sharding (inner pipelines judged recursively, inner chunks positive, rank-matched, and dividing the enclosing chunk); struct field rules. Codec chains are judged against the array each codec receives, so a shard behind a transpose must divide the transposed chunk and abytescodec behind acast_valueneeds an endianness for the target type. Rules are registered where they are defined; rules about one codec, grid, or data type live with that entity underrules._entitiesand are dispatched by name.validate_*/is_*/parse_*/check_*for array and group documents in both formats, structure and composition reported together.check_*returnsValid[T] | Invalid.v3._shape: one exact validator per known entity, key sets derived from the TypedDicts'__annotations__/__required_keys__. Unknown configuration members report as their ownunknown_keykind and never suppress other findings.create_*factories inzarr_metadata.builder, one per public document TypedDict,**kwargs: Unpack[<TypedDict>], one combined raise. The strict on-disk factories arecreate_zarr_v2_zarray_json/create_zarr_v2_zgroup_json.Boundary changes
dimension_namesvsshapeand v2chunksvsshapemove from the structural validator into rules.modelparsers and dataclasses now accept those documents.bytesinside shardingindex_codecsis rejected: the shard index isuint64, so the endianness is required there too.Relative to #296
Dropped: the incremental builder, the extension-point provenance table, and
TypeIscodec guards (all in part 3); fill-value propagation through codec chains (no rule read it, soArraySpecis two fields andNonereplaces theUNKNOWNsentinel); registry introspection helpers with no non-test consumer; a duplicated v2 consolidated envelope check. README and docs no longer claim the model layer checks dimension counts.Known follow-ups: entity rules do not yet recurse into nested
data_typepositions (struct fields,cast_valuetargets); sharding under a rectilinear grid is unjudged; v2 fill-value/dtype consistency has no rule.Verified: ruff, pyright 1.1.404 strict (0 errors), 752 tests, strict docs build.
🤖 Generated with Claude Code