feat(starrocks): carry consumed common-expr slots past the project that computes them - #1713
Draft
aocsa wants to merge 1 commit into
Draft
Conversation
…at computes them TPC-H q14 computes the revenue expression once as a common slot in the project below the aggregate and reads it from both measures. StarRocks' BE outputs a project's common slots whenever an ancestor references them; the translator now reproduces that by emitting the consumed slots as trailing columns past the descriptor row and carrying the (tuple, slot) -> column binding upward. A carried slot entering a join is refused, and the fragment root narrows the row back to the descriptor layout before names are derived. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Draft
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Layer 5 of the translator stack; base
stacked/translator-avg-expansion.TPC-H q14 is
100.00 * sum(case when p_type like 'PROMO%' then l_extendedprice * (1 - l_discount) else 0 end) / sum(l_extendedprice * (1 - l_discount)). The FE computes the revenue expression once as a common slot in the PROJECT_NODE below the aggregate and references that slot from both measures. No output tuple materializes it. On the stack below this layer the fragment is refused with a descriptor error, because #1233 resolves a common slot only inside the project that owns it. Q1 and Q6 do not hit this shape and do not need this layer.StarRocks' BE outputs a project's common slots whenever a node above references them, and the FE plans against that (it even emits a
slot_mapentry for the consumed slot). This layer reproduces that behavior.What changed, as one unit:
common_slots_consumed_aboveis a pre-pass over the flat preorder plan. For every PROJECT_NODE with a non-materialized common slot it scans the expression payloads of the strict ancestors, plus the fragment'soutput_exprs, for references to that slot. Matching is by slot id alone, because an ancestor ref can carry a stale tuple id.translate_plangains aroot_output_exprsparameter for this;lib.rspassesfragment.output_exprs.translate_project_nodeemits each consumed slot as a trailing column past the descriptor row (preferring the FE's ownslot_mapentry, falling back to the common binding) and records it as aCarriedSlot { tuple_id, slot_id, column }on the newTranslatedRel.carried_slots.[0, output_width - carried.len())are the descriptor row, the carried columns occupy the tail. Every construction site now decides what happens to carried columns. Filters, fetches, sorts without a sort tuple andappend_projectpass them through. Aggregates, sort-tuple projections, scans, exchange reads andemit_columnsstart from an empty list because they materialize a fresh row.ExprContext::with_carriedand the extendedtranslate_slot_refladder: exact synthetic override, exact carried(tuple, slot), descriptorslot_global_index, and only when the descriptor fails a carried column matched by slot id alone. A key that matches both an override and a carried column is refused; several by-slot-id candidates are refused. I preferred refusing over guessing everywhere, because a wrong column read is not an error the engine would catch.refuse_carried_join_child: a carried slot entering a hash or nested-loop join is refused. Join conjuncts resolve over the concatenated child rows using descriptor widths, so a wider left child would shift every right-side column while the descriptor math would not.confine_carriedat the fragment root, called inlib.rsbefore output names are derived. Carried columns are fragment-internal. Without this a width-preserving root (a SELECT conjunct over the carrying project) would ship columns that neither the sender's names nor the receiver's declared stream schema describe.Tests: 8 new integration tests in
tests/translate.rs, among themconsumed_common_slot_is_carried_to_the_aggregate,carried_common_slot_entering_a_join_is_refusedandroot_filter_over_a_carried_common_slot_is_narrowed_at_the_root, and 3 unit tests for the resolution ladder inexpr_translator.rs. The#1233tests on dev are unchanged and still pass, includingcommon_slots_outside_a_project_are_rejected;reject_common_slotsis untouched because no shape here needs a non-PROJECTcommon_slot_map.How I tested it. On a GB200 box (aarch64) I ran the CI trio: cargo fmt, clippy with warnings as errors, and the workspace test suite without the engine feature. All 224 tests pass, 11 of them new.
Not handled here: a consumer of a common slot across a join (refused, see above), and a consumer beyond an aggregate (fails with the existing descriptor error, pinned by
common_slot_consumed_beyond_the_aggregate_fails_loudly). Scan byte ranges and CN-side changes are separate stacks.Checklist
References