Skip to content

Prototype prepared indexing execution for codec consumers - #323

Draft
d-v-b wants to merge 2 commits into
zarr-indexing/grid-partitionfrom
codex/indexing-execution
Draft

Prototype prepared indexing execution for codec consumers#323
d-v-b wants to merge 2 commits into
zarr-indexing/grid-partitionfrom
codex/indexing-execution

Conversation

@d-v-b

@d-v-b d-v-b commented Sep 5, 2026

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

This draft adds an opt-in prepared execution layer on top of zarr-indexing/grid-partition. It lets codec consumers obtain chunk-local and result selectors directly from affine runs or connected-component tables, while retaining declarative projection lowering for other transforms. It does not replace Zarr's default indexers.

Preparation makes snapshot/borrow ownership and read/write conflict policies explicit. NumPy and shard consumers lower the same work to their respective selector layouts. The selection-flow guide remains on the base branch; this draft separately documents the experimental execution contracts and current sharding materialization boundary.

Review corrections included here:

  • Codec integration tests live in the package suite and skip only when Zarr is unavailable. The package CI runs them in the repo-root environment. Both pipelines cover v2, v3, and sharded arrays.
  • Complete boundary writes use the valid data extent. Storage-read assertions verify that full boundary writes, including stride-2 singleton selections, skip read-modify-write. Reverse selectors remain conservative because the codec shortcut also assumes buffer-order values.
  • Orthogonal scalar axes are removed before applying the remaining orthogonal selection. Negative scalars retain literal-coordinate bounds behavior. Vectorized mixed-scalar normalization remains outside this prototype's Zarr compatibility claims.
  • The mypy variable-type collision is fixed. Benchmark documentation contains reproduction and interpretation guidance, without historical timing tables or release notes for a private API.

Validation: 1,550 package tests passed, including 40 real codec integration cases; Ruff and commit hooks passed; _execution.py passed mypy; Pyright reported zero errors (95 warnings); strict MkDocs build passed. No end-to-end storage speedup is claimed. Compact sharding consumption and broader dispatch remain future work.

d-v-b and others added 2 commits September 6, 2026 12:05
Use data-extent completeness, drop orthogonal scalar axes at the execution boundary, and run codec tests in the package CI suite. Keep ownership and write policies in this separate experimental review.

Assisted-by: Codex:GPT-6
The planner now rejects diagonals with ValueError and has no shared
projection walk, so execute_transform factors the plan up front and lets
that rejection surface at construction instead of on first iteration.
_axis_plan reuses chunk_resolution's data-extent helper rather than
carrying a copy.

Assisted-by: ClaudeCode:claude-fable-5-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@d-v-b
d-v-b force-pushed the codex/indexing-execution branch from 1f2b4bc to dc2e279 Compare September 6, 2026 10:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant