Skip to content

Add capability to evaluate field-level type checks#2794

Open
albertlockett wants to merge 6 commits intoopen-telemetry:mainfrom
albertlockett:albert/opl-type-check-part2
Open

Add capability to evaluate field-level type checks#2794
albertlockett wants to merge 6 commits intoopen-telemetry:mainfrom
albertlockett:albert/opl-type-check-part2

Conversation

@albertlockett
Copy link
Copy Markdown
Member

Change Summary

Part 2 of what was started in #2754 ...

Adds the capability to OPL and the OTAP query-engine to evaluate logical expressions that check the type of some field. For example:

logs | where attributes["x"] is String
logs | where attributes["y"] is Integer

This i useful for cases where users may wish to do something with a field of unknown type (an AnyValue) and the operation would fail if the value was not the correct type. For example, maybe I wish to redact log body only if it is text, when I know that sometimes the body will actually be an int:

logs | if (body is String) {
  set body = concat(substring(body, 0, 3), "****") // <-- substring only accepts string type
}

In this case we use the GetType expression which is only supposed to return a value from the ValueType enum (see discussion here: #2754 (comment)), so currently only these values are supported. Although there are in fact many other types supported by OTel (e.g. specific int types like uint32, some etc.) the set of types in ValueType seem to be enough for most useful expressions, at least in the context of what the engine currently supports.

What issue does this PR close?

How are these changes tested?

Unit tests

Are there any user-facing changes?

Yes - this type of expressions is now available in transform processor.

@github-actions github-actions Bot added rust Pull requests that update Rust code query-engine Query Engine / Transform related tasks query-engine-columnar Columnar query engine which uses DataFusion to process OTAP Batches opl-parser Work items related to OPL Parser labels Apr 30, 2026
@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 30, 2026

Codecov Report

❌ Patch coverage is 94.00000% with 33 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.10%. Comparing base (524b78d) to head (1ec0cac).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2794      +/-   ##
==========================================
+ Coverage   86.09%   86.10%   +0.01%     
==========================================
  Files         694      695       +1     
  Lines      263421   263941     +520     
==========================================
+ Hits       226789   227273     +484     
- Misses      36108    36144      +36     
  Partials      524      524              
Components Coverage Δ
otap-dataflow 87.07% <96.38%> (+0.02%) ⬆️
query_abstraction 80.61% <ø> (ø)
query_engine 90.73% <41.66%> (-0.04%) ⬇️
otel-arrow-go 52.45% <ø> (ø)
quiver 92.25% <ø> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@albertlockett albertlockett marked this pull request as ready for review April 30, 2026 19:03
@albertlockett albertlockett requested a review from a team as a code owner April 30, 2026 19:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

opl-parser Work items related to OPL Parser query-engine Query Engine / Transform related tasks query-engine-columnar Columnar query engine which uses DataFusion to process OTAP Batches rust Pull requests that update Rust code

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

Support logical expressions that check the type of fields or the stream elements

1 participant