feat(starrocks): translate EXCHANGE_NODE as a stream read of the engine's exchange view - #1708
Draft
aocsa wants to merge 1 commit into
Draft
feat(starrocks): translate EXCHANGE_NODE as a stream read of the engine's exchange view#1708aocsa wants to merge 1 commit into
aocsa wants to merge 1 commit into
Conversation
…ne's exchange view An EXCHANGE_NODE is a fragment boundary and dev refuses it, so no multi-fragment plan translates. This lowers the receiver's exchange to a ReadRel over the engine's sirius_stream_<node_id> view, bound per node by the compute node through ExchangeInput, and records each stream's schema on TranslatedPlan.stream_inputs so the CN can declare it. Sender names bind to the row positionally; a merging exchange adds a SortRel. A hash-partitioned sink's bare slot keys resolve to output column indices. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Draft
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Layer 1 of the translator stack; base
dev.An
EXCHANGE_NODEis a fragment boundary.devrefuses it, so every multi-fragment plan fails to translate: every two-phase aggregate, every distributed join. TPC-H runs today only withSET new_planner_agg_stage = 1on one GPU.This layer lowers the receiver's exchange to a
ReadRelover the engine'ssirius_stream_<node_id>view. That name is the one the CN and the engine already share (sirius::ffi::stream_view_name, Rust binding in #1702). The compute node supplies, per exchange node, anExchangeInput { node_id, stream_view, names }through the newPlanTranslator::translate_fragment_with_exchange_inputs;translate_fragmentdelegates with no inputs and keeps refusing exchanges. A stream has no file to infer a schema from, so the translator records each exchange's schema (name plus DuckDB type name per column, derived from the sameTypethe read carries) onTranslatedPlan.stream_inputsfor the CN to declare.Rows never leave the GPU (
relay_fromon one node, packed bytes across nodes). That is why I did not take the materialize-to-parquet route of #1242; it added a GPU to file to GPU round trip per hop and could not name a remote sender.What the exchange becomes:
input_row_tuples(named_struct_for_tuples, which spans several tuples). Its column names are the sender's, bound by position: one name per column of the row layout, or the translation fails with a descriptor error. The receiver's own tuple still names the fragment root.sort_infopresent) becomes aSortRelover the stream read, built with the existing sort helpers. The senders' runs arrive in no fixed interleaving, so a plain read would drop the cross-fragment ORDER BY.offsetreachesapply_fetch, soOFFSET non an exchange emits aFetchRelwith an explicit unlimited count instead of being ignored.input_row_tuples, an emptystream_view, and a names/width mismatch.Two pieces the read needs ride along.
slot_global_indexfalls back to the slot id when exactly one row tuple carries it (the FE leaves grouping refs bound to the tuple below a multi-stage aggregation, the TPC-H q16 shape); two candidates is still an error. AndTranslatedPlan.output_partition_columnsresolves a HASH_PARTITIONED stream sink's keys to output column indices, bareSLOT_REFs only. A transformed key would make one sender hash a value its peers do not, so it is refused, as are a hash sink with no partition expressions and one combined withoutput_exprs. A root-level guard now rejects any width/name drift between the emitted row and its names.The CN crate changes are mechanical: three test literals gain the two new
TranslatedPlanfields.How I tested it. On a GB200 box (aarch64) I ran the CI trio: cargo fmt, clippy with warnings as errors, and the CN workspace tests without the engine feature. All 184 tests pass; the translator integration suite goes from 116 on dev to 127, and the 11 new ones (among them
merging_exchange_becomes_sort_over_stream_read,exchange_offset_becomes_a_fetchandhash_partitioned_sink_with_a_transformed_key_is_rejected) plus 2 new descriptor_table unit tests all pass.Not here: aggregation phases (partial/merge translation is the next layer; this one still accepts one-phase aggregates only) and the CN wiring that declares the streams and routes batches (the
cnstack).Checklist
References
translate_fragment_with_exchange_inputs/ExchangeInputAPI and three guard tests survive here, rewritten for streams.