Preserve BETWEEN range constraints without LHS re-eval by kumarUjjawal · Pull Request #5411 · tursodatabase/turso

kumarUjjawal · 2026-02-18T16:31:41Z

Description

Keep BETWEEN predicates indexable by extracting range constraints in the optimizer (>= start and <= end) without rewriting the AST.
Handle BETWEEN constraints in constraint collation/affinity resolution and consumption logic.
Add a snapshot test to verify BETWEEN with a subquery LHS is evaluated once.

Motivation and context

BETWEEN previously rewrote to two binary comparisons, which duplicated the LHS expression and could re-run expensive subqueries per row. The new translation evaluates LHS once, but we still need optimizer-level range constraints for SQLite compatibility and index usage. This change restores range planning while preserving single-evaluation semantics, and adds
a regression snapshot test.

Closes #5152

Description of AI Usage

Had a pair programming session with codex, the pr body is also written by codex.

turso-bot

Please review @jussisaurio

kumarUjjawal · 2026-02-18T16:54:15Z

Few Thoughts:

I moved BETWEEN handling into translation to avoid double evaluating the LHS (especially subqueries). That removed the old AST rewrite.
This caused a planner regression: the optimizer only recognized binary comparisons/IN, so BETWEEN stopped contributing range constraints and index seeks regressed.
I chose to keep BETWEEN intact in the AST and add optimizer-side constraint extraction instead. This keeps planning concerns in the optimizer and preserves single-evaluation semantics at execution.
While implementing this, I had two options: constraints expect binary expressions, and WHERE terms are consumed once. I updated constraining-expression resolution to understand BETWEEN, and only consume a BETWEEN term when both bounds are used.
NOT BETWEEN is not yet expanded into OR constraints for multi-index seeks; that remains a follow-up if we want parity there.
BETWEEN no longer being split into two separate AND terms. If we want to preserve the old multi-index AND plan, we can add a follow-up optimization to treat BETWEEN as two AND-able range constraints only for multi-index analysis

Any feedback on this would be much appreciated.

jussisaurio · 2026-02-20T12:38:12Z

core/translate/expr.rs

        }
-        ast::Expr::Between { .. } => {
-            crate::bail_parse_error!("BETWEEN expression should have been rewritten in optmizer")
+        ast::Expr::Between {


any way to share code between this and translate_expr?

jussisaurio · 2026-02-20T12:39:53Z

core/translate/expr.rs

+            program.reset_collation();
+
+            // Resolve collation for the first comparison (start vs lhs).
+            let lower_collation_ctx = resolve_collation_ctx(start_collation_ctx, lhs_collation_ctx);


can get_collseq_from_expr be used for this?

get_collseq_from_expr returns only a CollationSeq and loses the “explicit COLLATE vs column-derived” signal used to implement SQLite’s precedence rules.

jussisaurio · 2026-02-20T12:40:14Z

core/translate/expr.rs

-                        ast::Expr::Binary(Box::new(lower), ast::Operator::And, Box::new(upper))
-                    };
-                }
+                // Between is handled natively in translate_expr/translate_condition_expr,


This comment doesn't need to exist

jussisaurio · 2026-02-20T12:43:09Z

core/translate/optimizer/constraints.rs

+                (
+                    self.operator
+                        .as_ast_operator()
+                        .expect("expected an ast operator for BETWEEN constraint"),


which.....?

jussisaurio · 2026-02-20T12:43:48Z

core/translate/optimizer/constraints.rs

+                if let ast::Expr::Between { lhs, .. } = &where_term.expr {
+                    lhs.as_ref()
+                } else {
+                    panic!("Expected a valid binary expression");


Why are we deconstructing &where_term.expr twice? What is this panic message?

jussisaurio · 2026-02-20T12:46:21Z

core/translate/optimizer/mod.rs

                    let defer_cross_table_constraints =
                        hash_join_build_only_tables.contains(&table_idx);
+                    let mut used_constraints_per_term: HashMap<usize, usize> = HashMap::default();
+                    for cref in constraint_refs.iter() {


what is being done here?

We track how many constraints from each WHERE term are used so we only consume the term once (and only if both bounds are used). Added comments for explaination.

jussisaurio · 2026-02-20T12:46:39Z

core/translate/optimizer/mod.rs

-                                "trying to consume a where clause term twice: {where_term:?}",
-                            );
+                            if where_term.consumed {
+                                if matches!(where_term.expr, ast::Expr::Between { .. }) {


why this specialcasing?

it prevents double-consumption of the same WHERE term and avoids dropping the predicate when only one bound is used by the index.

kumarUjjawal requested review from jussisaurio and penberg as code owners February 18, 2026 16:31

turso-bot bot reviewed Feb 18, 2026

View reviewed changes

github-actions bot added core optimizer translation/planning labels Feb 18, 2026

jussisaurio reviewed Feb 20, 2026

View reviewed changes

kumarUjjawal added 3 commits February 20, 2026 19:58

Preserve BETWEEN range constraints without LHS re-eval

ba844c6

update tpch snapshot

78a4fa7

refactor for translate_between_lhs_start and translate_between_end

0cf6237

kumarUjjawal force-pushed the fix/between_evaluate_lhs branch from 8f14201 to 0cf6237 Compare February 20, 2026 14:42

fix fmt

cca5fcb

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

Preserve BETWEEN range constraints without LHS re-eval#5411

Preserve BETWEEN range constraints without LHS re-eval#5411
kumarUjjawal wants to merge 4 commits intotursodatabase:mainfrom
kumarUjjawal:fix/between_evaluate_lhs

kumarUjjawal commented Feb 18, 2026

Uh oh!

turso-bot bot left a comment

Uh oh!

kumarUjjawal commented Feb 18, 2026

Uh oh!

jussisaurio Feb 20, 2026

Uh oh!

jussisaurio Feb 20, 2026

Uh oh!

kumarUjjawal Feb 20, 2026

Uh oh!

jussisaurio Feb 20, 2026

Uh oh!

jussisaurio Feb 20, 2026

Uh oh!

jussisaurio Feb 20, 2026

Uh oh!

jussisaurio Feb 20, 2026

Uh oh!

kumarUjjawal Feb 20, 2026

Uh oh!

jussisaurio Feb 20, 2026

Uh oh!

kumarUjjawal Feb 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Comments

Conversation

kumarUjjawal commented Feb 18, 2026

Description

Motivation and context

Description of AI Usage

Uh oh!

turso-bot bot left a comment

Choose a reason for hiding this comment

Uh oh!

kumarUjjawal commented Feb 18, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants