OTTL types and compilation optimizations

[sfc-gh-bdrutu]

Component(s)

pkg/ottl

Is your feature request related to a problem? Please describe.

This is an umbrella issue that describes changes necessary to make OTTL type safe as well as allow to optimize the compiled code, here are just some examples:

• Remove redundant executions, for example attributes["foo"] != nil or attributes["foo"] == "bar" can be optimized to avoid double attributes lookup.
• Remove boxing/unboxing of the values to any type, for example I've seen a significant percentage >5% usage caused by runtime.convTstring.
• Determine type conversions errors during compilation and optimize code to avoid type checks during runtime.

Describe the solution you'd like

The solution requires changing a bit how OTTL works, but here is a rough timeline/roadmap (not necessary everything in the same order):

1. Change Expression and the Factory (function factory) so that we include information about the return type and enforce all incoming arguments are concrete types (e.g. avoid usage of the generic Getter). This way we know exactly the expected type as well as the returned type which is used possibly by other functions, for example IsString(attributes["foo"] == "bool" we can validate and optimize knowing that IsString returns a bool to avoid runtime type checks for the equality operator.
2. Change Getter and add Factory (factory for context Getters) so that we include information about the return type, same optimizations as 1.
3. Change where the path "keys" part is resolved. Right now this is resolved as part of the PathExpressionParser, but would be good to be a framework support (remove the context specific part of it), so that the framework can identify if same attribute is used twice to not look it up twice.
4. Once the first 3 publicly breaking changes land, start by changing our compilation step to initially create an intermediate code (AST) that we can iterate over and optimize execution since now we have all known information about types and informations required to optimize execution.
5. Add internal "context stack" to allow saving intermediate results to remove double execution of the same code, in the attributes retrieval case or any other complex computations.

Some nice to haves that may not be required but need to validate:

1. Move "set" to be part of the language instead of a function to allow more optimizations, example to understand when something changes in the context.

Describe alternatives you've considered

I've consider a complete re-write of the OTTL (v2) package, but that will be hard to deliver and land.

Additional context

No response

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[github-actions]

Pinging code owners for pkg/ottl: @TylerHelmuth @evan-bradley @edmocosta @bogdandrutu. See Adding Labels via Comments if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triage' to set the priority and remove the needs-triage label.

[braydonk]

@aviralgarg05, @itssaharsh and future GSoC contributors: Please keep discussion off of this issue. This issue may or may not end up being part of the GSoC Project, but we need to keep this issue directed toward the discussion on the actual issue at hand and not on GSoC proposals. We have received the proposals but due to the high volume we haven't been able to respond to all of them yet. We're going to come up with a unified place to discuss GSoC, but let's not do that here.

I will remove the GSoC related comments that exist here already, but your interest is noted and I will reach out when we have a formal place to submit proposals.

[rite7sh]

Hi! I’ve been exploring the OTTL type system improvements mentioned in this issue.
One potential first step toward enabling compile-time type information would be to add return type metadata to Expr. The idea would be to extend the Expr struct with a returnType field (defaulting to TypeUnknown) and expose a ReturnType() method. This would not change existing behavior but would allow future work on compile-time validation and optimization.
Would a small infrastructure PR introducing this metadata be acceptable as a starting point?