feat(codegen): Substrait front-end for the codegen serving engine - #1
Closed
ahrzb wants to merge 2 commits into
Closed
feat(codegen): Substrait front-end for the codegen serving engine#1ahrzb wants to merge 2 commits into
ahrzb wants to merge 2 commits into
Conversation
* ci: review PRs with Claude Code + superpowers plugin Runs anthropics/claude-code-action on pull_request, installing the superpowers plugin via the action's plugin_marketplaces/plugins inputs. Requires a CLAUDE_CODE_OAUTH_TOKEN repo secret. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * backlog: prioritize m-1, rule decision-7 (native default, codegen opt-in) - m-1 priorities set: High spine (TASK-5/6/13), Medium (TASK-2/3/15/16), Low (TASK-14/17) - decision-7 accepted + renamed: native is the default engine, codegen is opt-in; TASK-29/34 declassified from blocked-on-open to decided-Low - Closed: TASK-27/30 (CASE both engines), TASK-28 (re-scoped to match-DataFusion folding + retitled), TASK-31, TASK-32, TASK-7 (suite now 0 xfailed) - New: TASK-31 (CASE-branch transformer test), TASK-32 (pydantic-v3 probe), TASK-33 (stale .pyd rebuild guard), TASK-34 (codegen transformer support) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: add serving-path benchmark brief (doc-6) Benchmarks the serving-path options -- ONNX vs the native Rust interpreter vs Python codegen vs WASM. Authored in the benchmarking/QA session (Fermi); committed here at AmirHossein's request. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: grant id-token:write for Claude review OIDC + bump checkout to v5 claude-code-action needs id-token:write to fetch its OIDC token — without it the run failed 'Could not fetch an OIDC token'. Also bump actions/checkout v4->v5 (Node 20 deprecation). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Add CodegenSubstraitFn, a second codegen backend that consumes a Substrait plan instead of SQL. The plan is translated into the existing _codegen plan IR and handed to the same backend-agnostic pipeline (optimize/validate/infer/ compile) and rt.* runtime — only the front-end differs. - _codegen_substrait/: Context (row schemas + static tables), consumer (Substrait -> _codegen IR: Read/Filter/Project, scalar exprs, positional field refs, function anchors, literals, casts), and CodegenSubstraitFn. - Reuse: extract CodegenFn._finalize so both the SQL and Substrait front-ends join at the plan IR; nothing downstream changes. - Input is a Substrait plan (bytes); from_sql() produces one via DataFusion for callers/tests. Table names are resolved case-insensitively (DataFusion folds them). Surfaces DataFusion can't serialize (e.g. UNNEST) defer cleanly. - Wire a third "substrait" backend into the differential harness; it passes the supported scalar/scan/filter/project surface and skips the rest. Scope: the basic relational/scalar surface. Joins, string builtins, CASE and transformers are deferred (skip, not fail). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ahrzb
force-pushed
the
feat/codegen-substrait-backend
branch
from
July 22, 2026 00:25
5669713 to
3330362
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
CodegenSubstraitFn— a second codegen backend that takes a Substrait plan as input instead of SQL. The plan is translated into the existing_codegenplan IR, then handed to the same backend-agnostic pipeline (optimize/validate/infer_type/compile) and thert.*runtime. Only the front-end differs.Design
_codegen_substrait/—Context(registered row-table schemas + static tables),consumer(Substrait → IR: Read/Filter/top-level Project; field references resolved from positional indices; function anchors; literals; casts), andCodegenSubstraitFn.CodegenFn._finalizeso the SQL and Substrait front-ends join at the plan IR — nothing downstream duplicated or changed.from_sql()produces one via DataFusion (registering theContext's tables) for callers and tests — so a hand-supplied plan and SQL are one code path.UNNEST) defer cleanly rather than crash.Scope (intentionally "the basic part")
Scan · project · scalar expressions (arithmetic / comparison / boolean / a few builtins) · filter · cast. Deferred (skip, not fail): joins, string builtins,
CASE, and transformers/{ref}callouts.Testing
substraitbackend into the existing differential harness — it runs the full oracle-checked suite, passing the supported surface and skipping the rest.engine_test.pypins the raw-plan-bytes entry point and clean deferral.Notes
substrait(protobuf bindings) — added topyproject.toml;datafusion(the producer) was already a dep.InferFnremains the parity oracle via the differential harness.🤖 Generated with Claude Code