Skip to content

fix: coerce SIMILAR TO operands to a common string type#23704

Open
u70b3 wants to merge 1 commit into
apache:mainfrom
u70b3:fix/similar-to-type-coercion
Open

fix: coerce SIMILAR TO operands to a common string type#23704
u70b3 wants to merge 1 commit into
apache:mainfrom
u70b3:fix/similar-to-type-coercion

Conversation

@u70b3

@u70b3 u70b3 commented Jul 20, 2026

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

SIMILAR TO panics with failed to downcast array whenever its operands end up as arrays of different physical types:

SELECT 'a' SIMILAR TO NULL;  -- panics during planning (constant folding)

CREATE TABLE t AS SELECT * FROM (VALUES ('user auth failed')) v(s);
CREATE TABLE p AS SELECT * FROM (VALUES ('(auth|login)')) v(pat);
SELECT arrow_cast(t.s, 'Utf8View') SIMILAR TO p.pat FROM t CROSS JOIN p;  -- panics at runtime

Root cause: SIMILAR TO is planned as a regex binary operator (RegexMatch et al.), whose kernel downcasts both arrays to the left operand's array type. But unlike LIKE and the regex operators (~, ~*, ...), the TypeCoercion analyzer never coerced Expr::SimilarTo operands — it was listed in the "nothing to coerce" arm of TypeCoercionRewriter — so any type mismatch reached the kernel un-normalized and hit a blind .expect(). Literal patterns take a scalar fast path, which is why the common col SIMILAR TO 'pattern' form never showed this.

What changes are included in this PR?

Two commits:

  1. Coerce SIMILAR TO operands in the analyzer (datafusion/optimizer/src/analyzer/type_coercion.rs)

    • Extract the existing LIKE-operand coercion into coerce_like_operands (behavior for LIKE/ILIKE is unchanged, including the Dictionary(_, Utf8) special case and error texts).
    • Apply regex_coercion to Expr::SimilarTo — the same coercion the physical regex operators it is planned into use (expr-common/src/type_coercion/binary.rs:218). A NULL pattern is coerced to a typed NULL and evaluates to NULL (matching expr ~ NULL), and mixed string types are unified before planning.
    • Regression coverage in type_coercion.slt (both reproducers, NULL variants, and the planning-error path) plus analyzer unit tests.
  2. Defense in depth in the regex kernels (datafusion/physical-expr/src/expressions/binary/kernels.rs)

    • Replace .expect("failed to downcast array") in regexp_is_match_flag! / regexp_is_match_flag_scalar! with exec_err!, so any expression path that bypasses the analyzer (e.g. a hand-built plan going straight to physical planning) returns an error instead of panicking. Includes a unit test constructing a mismatched RegexMatch directly.

Are these changes tested?

Yes:

  • sqllogictest: 7 new cases in type_coercion.slt (issue reproducers, NULL/NOT SIMILAR TO NULL, LargeUtf8 × Utf8, incompatible-type planning error).
  • Unit tests: analyzer coercion plans (NULL pattern, cross string types, error case) and a kernel-level test proving mismatched arrays now error instead of panic.
  • Verified ./dev/rust_lint.sh, cargo test -p datafusion-optimizer --lib, cargo test -p datafusion-physical-expr --lib, and the relevant sqllogictest files (type_coercion, strings, scalar, regexp, string) — all green.

Are there any user-facing changes?

Only the bug fix: queries that previously panicked now either evaluate correctly (Utf8View × Utf8 etc.) or return NULL / a planning error. LIKE/ILIKE behavior is unchanged.

@codecov-commenter

codecov-commenter commented Jul 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.66667% with 20 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.67%. Comparing base (67947b6) to head (54f17f7).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/optimizer/src/analyzer/type_coercion.rs 87.96% 6 Missing and 10 partials ⚠️
datafusion/physical-expr/src/expressions/binary.rs 76.47% 1 Missing and 3 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #23704      +/-   ##
==========================================
- Coverage   80.67%   80.67%   -0.01%     
==========================================
  Files        1088     1088              
  Lines      367591   367729     +138     
  Branches   367591   367729     +138     
==========================================
+ Hits       296563   296673     +110     
- Misses      53374    53386      +12     
- Partials    17654    17670      +16     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

SIMILAR TO is planned as a regex binary operator, so normalize its operands with regex coercion in the analyzer. This handles typed NULLs, mixed UTF8 physical types, and dictionary values before physical planning while preserving LIKE dictionary behavior.

Replace blind regex kernel downcasts with execution errors for plans that bypass the analyzer, and cover planner and runtime cases including dictionary arrays.

Closes apache#22886.
@u70b3
u70b3 force-pushed the fix/similar-to-type-coercion branch from dfdf367 to 54f17f7 Compare July 20, 2026 03:26
Comment on lines +446 to +449
/// Coerce the value and pattern expressions of a string pattern matching
/// expression (`LIKE`, `ILIKE` or `SIMILAR TO`) to a common type using
/// the provided coercion rules.
fn coerce_like_operands(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a pure refactor 👍🏻

)
})?;
let expr = match left_type {
DataType::Dictionary(_, inner) if *inner == DataType::Utf8 => Box::new(expr),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is pre-existing and so I think it should be addressed separately but this does not handle dictionary encoded needles, i.e.

CREATE TABLE t AS SELECT * FROM (VALUES ('user auth failed')) v(s);
CREATE TABLE p AS SELECT * FROM (VALUES ('(auth|login)')) v(pat);
SELECT arrow_cast(t.s, 'Dictionary(Int32, Utf8)') SIMILAR TO p.pat FROM t CROSS JOIN p;

I think this can be handled as it's own issue.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review!

You're right it's pre-existing — I checked the neighboring behavior: the LIKE kernels handle Dictionary(_, Utf8) needles in the array-array path, but the regex kernels (regex_match_dyn) only handle dictionaries in the scalar fast path. With this PR those paths now return a proper error instead of panicking, but dictionary support itself is indeed a separate concern.

Filed #23709 to track it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optimizer Optimizer rules physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SIMILAR TO panics ('failed to downcast array') when operand types differ (e.g. NULL pattern, Utf8View vs Utf8)

3 participants