-
Notifications
You must be signed in to change notification settings - Fork 3.3k
Description
Component(s)
connector/routing
Is your feature request related to a problem? Please describe.
This is regarding a recently added feature via #45061
I am using the following config and I can notice that when action: copy is used under the pipelines, the data is forwarded to default pipelines as well. Now, I want data to be forwarded to other pipelines but in case the statements are evaluated to be true, I don't want data to be routed to default pipeline. In case, one or more conditions are matched, I want data to only go through non-default-pipelines.
`
connectors:
routing/logs:
default_pipelines: [logs/file/default]
table:
- statement: route() where resource.attributes["service.name"] == "demo-service"
pipelines: [logs/file/demo-service]
action: copy
- statement: route() where resource.attributes["env"] == "dev"
pipelines: [ logs/file/env ]
action: copy`
For example,
if resource.attributes = {service.name: demo-service, env: dev}, I see data going to all three pipelines (logs/file/default, logs/file-demo-serivce & logs/file/env). But since there are one or more match present in the table entries, I don't want data to go to default pipeline.
Is there any way to achieve this functionality with the existing configurations?
Describe the solution you'd like
If such functionality does not exist today, can we add a new action: copyToNonDefault config option which achieves this? If this config is used, and one or more condition is matched then data would not be routed to the default pipelines.
If zero condition is matched then and only then data would be routed to default pipelines.
Describe alternatives you've considered
Earlier, we were using routing processor (which used to evaluate more than one routing conditions) but since it was removed from the otel collector, we have tried using multi-layered routing connector approach listed here: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/connector/routingconnector/README.md#example-with-default_pipelines
It has been helpful but lately we have noticed that a lot of complex routing conditions are hard to write since routing connector executes the first matching condition only. The configuration has become extremely verbose and the amount of data being copied to all the pipelines is hurting the performance as well.
Since the new actionconfig has been doing what we need to achieve, it would be great to have a new option under it to not forward data to non-default pipelines.
Additional context
No response
Tip
React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.