Skip to content

[fix] correct filter using expressions where each side returns nulls and/or mixed types#2758

Open
albertlockett wants to merge 2 commits intoopen-telemetry:mainfrom
albertlockett:albert/2659
Open

[fix] correct filter using expressions where each side returns nulls and/or mixed types#2758
albertlockett wants to merge 2 commits intoopen-telemetry:mainfrom
albertlockett:albert/2659

Conversation

@albertlockett
Copy link
Copy Markdown
Member

Change Summary

Corrects the behaviour of filtering expressions where each side of a relative expression returns either nulls, types that are not equivalent and/or struct columns representing AnyValues.

For example, in a program like this:

logs | where attributes["x"] == attributes["y"]

In this case, we don't know at compile time what is the type of the left or right side of the == expression, so we evaluate the expressions on either side and then attempt to compare the results. Before this change, we were effectively just naively using the eq compute kernel (technically this would get called inside the BinaryExpr). This means that we'd produce incorrect results or errors in cases where:

  • the type on the left / right were not the same (we returned an error)
  • both sides were null (we return false, but null == nullshould return true)
  • if one side was a struct representing an AnyValue (such as log body, although attributes may also be coerced into a struct like this), we would also just return an error.

This PR changes the implementation to use a special compare kernel which handles all these cases. Effectively the logic it applies for each row would be:

  • if the types do not match, evaluate as false
  • null handling:
    • if both sides are null and the operator is ==, return true
    • if exactly one side is null and the operator is != return true
    • otherwise return false

The compare handles comparing the appropriate values columns for AnyValues represented as struct columns. It also makes the determination if one side is effectively a null for purpose of comparison -- it considers the row null when any of the following conditions are met:

  • the struct column is null
  • the value in the type column is Empty
  • the value columns elected by a non-Empty type column value is null

What issue does this PR close?

How are these changes tested?

Unit tests

Are there any user-facing changes?

No - this is basically just a bug fix for how OTAP query engine handles some filter expressions.

@github-actions github-actions Bot added rust Pull requests that update Rust code query-engine Query Engine / Transform related tasks query-engine-columnar Columnar query engine which uses DataFusion to process OTAP Batches labels Apr 25, 2026
@albertlockett albertlockett changed the title Filter using expressions where each side returns nulls and/or mixed types [fix] correct filter using expressions where each side returns nulls and/or mixed types Apr 25, 2026
@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 25, 2026

Codecov Report

❌ Patch coverage is 96.01113% with 43 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.16%. Comparing base (bc4cec8) to head (3909002).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2758      +/-   ##
==========================================
+ Coverage   88.13%   88.16%   +0.02%     
==========================================
  Files         645      646       +1     
  Lines      248445   249458    +1013     
==========================================
+ Hits       218956   219923     +967     
- Misses      28965    29011      +46     
  Partials      524      524              
Components Coverage Δ
otap-dataflow 89.74% <96.01%> (+0.02%) ⬆️
query_abstraction 80.61% <ø> (ø)
query_engine 90.75% <ø> (ø)
otel-arrow-go 52.45% <ø> (ø)
quiver 92.25% <ø> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@albertlockett albertlockett marked this pull request as ready for review April 25, 2026 23:04
@albertlockett albertlockett requested a review from a team as a code owner April 25, 2026 23:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

query-engine Query Engine / Transform related tasks query-engine-columnar Columnar query engine which uses DataFusion to process OTAP Batches rust Pull requests that update Rust code

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

Columnar Query engine comparing fields with different types/nulls does not have correct behaviour

1 participant