Conversation
|
This is a preview of the changelog of the next release. If this branch is not up-to-date with the current main branch, the changelog may not be accurate. Rebase your branch on the main branch to get the most accurate changelog. Note that this might contain changes that are on main, but not yet released. Changelog: 0.13.0 (2026-07-29)⚠ BREAKING CHANGES
Features |
There was a problem hiding this comment.
Pull request overview
Adds a new join pipeline operator to the SaneQL query engine, implemented as a two-child JoinNode backed by Arrow Acero’s hash join, and wires it through AST conversion and optimizer passes.
Changes:
- Introduces
JoinNode(schema + exec-plan generation + JSON plan output) and registersjoin(left, right, on, type := inner)in the SaneQL function registry. - Extends the optimizer framework and key passes (filter pushdown + column narrowing) to traverse/handle a two-child join node.
- Adds query-fixture-based tests covering several join scenarios and error cases.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| src/silo/query_engine/saneql/ast_to_query.cpp | Parses/validates join arguments, extracts join keys from on, parses join type, registers join() |
| src/silo/query_engine/optimizer/pipeline_pass_base.h | Adds default propagation for two-child JoinNode |
| src/silo/query_engine/optimizer/filter_pushdown_pass.{h,cpp} | Adds join-aware filter distribution/pushdown logic |
| src/silo/query_engine/optimizer/column_narrowing_pass.{h,cpp} | Ensures both join branches keep full schemas below join while still recursing |
| src/silo/query_engine/operators/{join_node.h,join_node.cpp} | Implements join operator node (schema, exec plan via hashjoin, plan JSON) |
| src/silo/query_engine/operators/query_node.{h,cpp} | Adds NodeKind::JOIN + stringification |
| src/silo/query_engine/operator_visitor.h | Enables visitor dispatch for JoinNode |
| src/silo/query_engine/operators/join_node.test.cpp | Adds end-to-end query scenarios for join behavior |
72f273f to
305561f
Compare
fengelniederhammer
left a comment
There was a problem hiding this comment.
Looks good! Just a few things.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 16 out of 16 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (3)
src/silo/query_engine/operators/join_node.h:25
- The JoinNode docstring says overlapping column names are passed through as-is, but SaneQL join parsing explicitly rejects overlapping names (ast_to_query.cpp:1549-1562) and the user docs describe this restriction. This comment is currently misleading and contradicts the implemented behavior.
/// The output schema is the left child's columns followed by the right child's
/// columns, except for semi/anti joins which output only the columns of the
/// side they keep. Overlapping column names between the two sides are passed
/// through as-is (mirroring SQL `SELECT *`); rename or project them downstream
/// if distinct names are required.
src/silo/query_engine/optimizer/filter_pushdown_pass.cpp:3
#include <vector>is unused in this translation unit (the file ends at line 138 and does not reference std::vector directly). Removing it avoids unnecessary includes.
#include <vector>
src/silo/query_engine/saneql/ast_to_query.cpp:1504
parseJoinType()callsextractIdentifierName(*type_expr)without a join-specific validation/error message. If a user passes a non-identifier (e.g. a string literal), they will get the generic "expected identifier" error from the AST helper, which is inconsistent with the other join() diagnostics in this file.
const auto* type_expr = args.get("type");
const auto name = extractIdentifierName(*type_expr);
if (name == "inner") {
resolves #1379
Summary
Add a
joinpipeline operator to the SaneQL query engine that joins twoinput pipelines on an equi-join condition:
onis an equality (or&&-conjunction of equalities) between a leftand a right column; the key pairs are extracted into an Arrow hash join.
typeselects the join kind: inner (default), left, right, full,leftSemi, rightSemi, leftAnti, rightAnti.
Implemented as a new two-child JoinNode operator backed by Arrow Acero's
"hashjoin" exec node. The optimizer passes are extended for the second
child: FilterPushdownPass rejects any predicate sitting above the join
(which input a predicate belongs to could be derived from freeIUs(), but
pushing into the null-supplying side of an outer join, or pushing a
predicate that references no column, would change the result -- apply
filters to the join inputs instead), and ColumnNarrowingPass keeps both
inputs' columns (including the join keys) intact below the join.
Known limitation: the two inputs must have disjoint column names.
Overlapping names would make column references ambiguous and produce a
result with duplicate column names, so such joins are rejected; rename
one side (e.g. via map()) or drop the duplicates (e.g. via project())
first.
PR Checklist