Skip to content

feat(query_engine): add join operator - #1385

Merged
taepper merged 2 commits into
mainfrom
1379-join
Jul 29, 2026
Merged

feat(query_engine): add join operator#1385
taepper merged 2 commits into
mainfrom
1379-join

Conversation

@taepper

@taepper taepper commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

resolves #1379

Summary

Add a join pipeline operator to the SaneQL query engine that joins two
input pipelines on an equi-join condition:

join(left, right, on, type := inner)
  • on is an equality (or &&-conjunction of equalities) between a left
    and a right column; the key pairs are extracted into an Arrow hash join.
  • type selects the join kind: inner (default), left, right, full,
    leftSemi, rightSemi, leftAnti, rightAnti.

Implemented as a new two-child JoinNode operator backed by Arrow Acero's
"hashjoin" exec node. The optimizer passes are extended for the second
child: FilterPushdownPass rejects any predicate sitting above the join
(which input a predicate belongs to could be derived from freeIUs(), but
pushing into the null-supplying side of an outer join, or pushing a
predicate that references no column, would change the result -- apply
filters to the join inputs instead), and ColumnNarrowingPass keeps both
inputs' columns (including the join keys) intact below the join.

Known limitation: the two inputs must have disjoint column names.
Overlapping names would make column references ambiguous and produce a
result with duplicate column names, so such joins are rejected; rename
one side (e.g. via map()) or drop the duplicates (e.g. via project())
first.

PR Checklist

  • All necessary documentation has been adapted or there is an issue to do so.
  • The implemented feature is covered by an appropriate test.

@github-actions

github-actions Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

This is a preview of the changelog of the next release. If this branch is not up-to-date with the current main branch, the changelog may not be accurate. Rebase your branch on the main branch to get the most accurate changelog.

Note that this might contain changes that are on main, but not yet released.

Changelog:

0.13.0 (2026-07-29)

⚠ BREAKING CHANGES

  • preprocessing: the database config options defaultNucleotideSequence and defaultAminoAcidSequence have been removed. sequenceName is now required on all nucleotide and amino acid filters (nucleotideEquals/aminoAcidEquals, hasMutation/hasAAMutation, insertionContains/aminoAcidInsertionContains, symbolInSet and the mutation profile filters).

Features

  • app: add a --version command to the silo binary (8b3146e)
  • preprocessing: remove default sequence names from database config (#1386) (b8712ec)
  • query_engine: add join operator (01aeb0d)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new join pipeline operator to the SaneQL query engine, implemented as a two-child JoinNode backed by Arrow Acero’s hash join, and wires it through AST conversion and optimizer passes.

Changes:

  • Introduces JoinNode (schema + exec-plan generation + JSON plan output) and registers join(left, right, on, type := inner) in the SaneQL function registry.
  • Extends the optimizer framework and key passes (filter pushdown + column narrowing) to traverse/handle a two-child join node.
  • Adds query-fixture-based tests covering several join scenarios and error cases.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
src/silo/query_engine/saneql/ast_to_query.cpp Parses/validates join arguments, extracts join keys from on, parses join type, registers join()
src/silo/query_engine/optimizer/pipeline_pass_base.h Adds default propagation for two-child JoinNode
src/silo/query_engine/optimizer/filter_pushdown_pass.{h,cpp} Adds join-aware filter distribution/pushdown logic
src/silo/query_engine/optimizer/column_narrowing_pass.{h,cpp} Ensures both join branches keep full schemas below join while still recursing
src/silo/query_engine/operators/{join_node.h,join_node.cpp} Implements join operator node (schema, exec plan via hashjoin, plan JSON)
src/silo/query_engine/operators/query_node.{h,cpp} Adds NodeKind::JOIN + stringification
src/silo/query_engine/operator_visitor.h Enables visitor dispatch for JoinNode
src/silo/query_engine/operators/join_node.test.cpp Adds end-to-end query scenarios for join behavior

Comment thread src/silo/query_engine/optimizer/filter_pushdown_pass.cpp Outdated
Comment thread src/silo/query_engine/operators/join_node.cpp Outdated
Comment thread src/silo/query_engine/saneql/ast_to_query.cpp
Comment thread src/silo/query_engine/operators/join_node.test.cpp

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 4 comments.

Comment thread src/silo/query_engine/saneql/ast_to_query.cpp
Comment thread src/silo/query_engine/optimizer/filter_pushdown_pass.cpp Outdated
Comment thread src/silo/query_engine/operators/join_node.h
Comment thread documentation/query_documentation.md Outdated

@fengelniederhammer fengelniederhammer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Just a few things.

Comment thread src/silo/query_engine/operators/join_node.cpp Outdated
Comment thread src/silo/query_engine/operators/join_node.cpp Outdated
Comment thread src/silo/query_engine/saneql/ast_to_query.cpp
Comment thread src/silo/query_engine/saneql/ast_to_query.cpp
Comment thread src/silo/query_engine/operators/join_node.cpp
Comment thread src/silo/query_engine/operators/join_node.test.cpp
@taepper taepper changed the title feat(engine): add join operator feat(query_engine): add join operator Jul 28, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

src/silo/query_engine/operators/join_node.h:25

  • The JoinNode docstring says overlapping column names are passed through as-is, but SaneQL join parsing explicitly rejects overlapping names (ast_to_query.cpp:1549-1562) and the user docs describe this restriction. This comment is currently misleading and contradicts the implemented behavior.
/// The output schema is the left child's columns followed by the right child's
/// columns, except for semi/anti joins which output only the columns of the
/// side they keep. Overlapping column names between the two sides are passed
/// through as-is (mirroring SQL `SELECT *`); rename or project them downstream
/// if distinct names are required.

src/silo/query_engine/optimizer/filter_pushdown_pass.cpp:3

  • #include <vector> is unused in this translation unit (the file ends at line 138 and does not reference std::vector directly). Removing it avoids unnecessary includes.
#include <vector>

src/silo/query_engine/saneql/ast_to_query.cpp:1504

  • parseJoinType() calls extractIdentifierName(*type_expr) without a join-specific validation/error message. If a user passes a non-identifier (e.g. a string literal), they will get the generic "expected identifier" error from the AST helper, which is inconsistent with the other join() diagnostics in this file.
   const auto* type_expr = args.get("type");
   const auto name = extractIdentifierName(*type_expr);
   if (name == "inner") {

@taepper
taepper merged commit 62990c7 into main Jul 29, 2026
20 checks passed
@taepper
taepper deleted the 1379-join branch July 29, 2026 07:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a join operator

3 participants