diff --git a/.chloggen/filterprocessor_context_inference_doc.yaml b/.chloggen/filterprocessor_context_inference_doc.yaml new file mode 100644 index 0000000000000..9a7c9e141cfc5 --- /dev/null +++ b/.chloggen/filterprocessor_context_inference_doc.yaml @@ -0,0 +1,31 @@ +# Use this changelog template to create an entry for release notes. + +# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix' +change_type: enhancement + +# The name of the component, or a single word describing the area of concern, (e.g. receiver/filelog) +component: processor/filter + +# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`). +note: Document new *_conditions configuration format with context inference + +# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists. +issues: [37904] + +# (Optional) One or more lines of additional information to render under the primary note. +# These lines will be padded with 2 spaces and then inserted directly into the document. +# Use pipe (|) for multiline entries. +subtext: | + Added documentation for the new Basic and Advanced configuration styles: + - Basic Config: flat list of OTTL conditions with automatic context inference + - Advanced Config: explicit context and per-group error_mode settings + - Legacy Configuration migration guide + +# If your change doesn't affect end users or the exported elements of any package, +# you should instead start your pull request title with [chore] or use the "Skip Changelog" label. +# Optional: The change log or logs in which this entry should be included. +# e.g. '[user]' or '[user, api]' +# Include 'user' if the change is relevant to end users. +# Include 'api' if there is a change to a library API. +# Default: '[user]' +change_logs: [user] diff --git a/processor/filterprocessor/README.md b/processor/filterprocessor/README.md index 7dabad2a6e215..da9091824cbc1 100644 --- a/processor/filterprocessor/README.md +++ b/processor/filterprocessor/README.md @@ -17,31 +17,52 @@ [k8s]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-k8s -The filterprocessor allows dropping spans, span events, metrics, datapoints, and logs from the collector. +The Filter Processor allows dropping spans, span events, metrics, datapoints, and logs from the collector. ## Configuration -The filterprocessor utilizes the [OpenTelemetry Transformation Language](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/README.md) +The Filter Processor utilizes the [OpenTelemetry Transformation Language](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/README.md) to create conditions that determine when telemetry should be dropped. If **any** condition is met, the telemetry is dropped (each condition is ORed together). -Each configuration option corresponds with a different type of telemetry and OTTL Context. -See the table below for details on each context and the fields it exposes. - -| Config | OTTL Context | -|---------------------|------------------------------------------------------------------------------------------------------------------------------------| -| `traces.span` | [Span](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/contexts/ottlspan/README.md) | -| `traces.spanevent` | [SpanEvent](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/contexts/ottlspanevent/README.md) | -| `metrics.metric` | [Metric](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/contexts/ottlmetric/README.md) | -| `metrics.datapoint` | [DataPoint](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/contexts/ottldatapoint/README.md) | -| `logs.log_record` | [Log](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/contexts/ottllog/README.md) | -| `profiles.profile` | [Profile](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/contexts/ottlprofile/README.md) | + +### General Config + +```yaml +filter: + error_mode: propagate + _conditions: [] +``` + +The Filter Processor's primary configuration is broken down by signal (traces, metrics, logs, and profiles) +and allows you to configure a list of conditions for the processor to evaluate. The list can be made of: + +- OTTL conditions. This option will meet most user's needs. See [Basic Config](#basic-config) for more details. +- Objects, which allows users to apply configuration options to a specific list of conditions. See [Advanced Config](#advanced-config) for more details. + +Within each `_conditions` list, only certain OTTL Contexts can be used. Each context provides access to different telemetry fields - click the context name for detailed documentation: + +| Signal | Available Contexts | +|--------------------|------------------------------------------------| +| trace_conditions | [resource], [scope], [span], and [spanevent] | +| metric_conditions | [resource], [scope], [metric], and [datapoint] | +| log_conditions | [resource], [scope], and [log] | +| profile_conditions | [resource], [scope], and [profile] | + +[resource]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/contexts/ottlresource/README.md +[scope]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/contexts/ottlscope/README.md +[span]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/contexts/ottlspan/README.md +[spanevent]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/contexts/ottlspanevent/README.md +[metric]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/contexts/ottlmetric/README.md +[datapoint]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/contexts/ottldatapoint/README.md +[log]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/contexts/ottllog/README.md +[profile]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/contexts/ottlprofile/README.md The OTTL allows the use of `and`, `or`, and `()` in conditions. See [OTTL Boolean Expressions](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/LANGUAGE.md#boolean-expressions) for more details. For conditions that apply to the same signal, such as spans and span events, if the "higher" level telemetry matches a condition and is dropped, the "lower" level condition will not be checked. This means that if a span is dropped but a span event condition was defined, the span event condition will not be checked for that span. -The same relationship applies to metrics and datapoints. +The same relationship applies to logs, metrics and datapoints. If all span events for a span are dropped, the span will be left intact. If all datapoints for a metric are dropped, the metric will also be dropped. @@ -56,44 +77,128 @@ The filter processor also allows configuring an optional field, `error_mode`, wh If not specified, `propagate` will be used. -### Examples +### Basic Config + +The basic configuration style allows you to configure OTTL conditions as a flat list, without worrying about extra configurations. + +**This is the simplest way to configure the Filter Processor.** If you need explicit context specification or specific error modes, see [Advanced Config](#advanced-config). + +Format: + +```yaml +filter: + error_mode: propagate + _conditions: + - string + - string + - string +``` + +Example: ```yaml processors: - filter/ottl: - error_mode: ignore - traces: - span: - - 'attributes["container.name"] == "app_container_1"' - - 'resource.attributes["host.name"] == "localhost"' - - 'name == "app_3"' - spanevent: - - 'attributes["grpc"] == true' - - 'IsMatch(name, ".*grpc.*")' - metrics: - metric: + filter: + error_mode: propagate + trace_conditions: + - 'span.attributes["container.name"] == "app_container_1"' + - 'resource.attributes["host.name"] == "localhost"' + - 'span.name == "app_3"' + metric_conditions: + - 'metric.name == "my.metric" and resource.attributes["my_label"] == "abc123"' + - 'metric.type == METRIC_DATA_TYPE_HISTOGRAM' + - 'resource.attributes["service.name"] == "my_service_name"' + log_conditions: + - 'IsMatch(log.body, ".*password.*")' + - 'log.severity_number < SEVERITY_NUMBER_WARN' + profile_conditions: + - 'profile.duration_nanos > 3000' +``` + +#### Context Inference + +The Filter Processor automatically infers the OTTL Context based on the paths used in a condition. +Each condition in a list is parsed individually and grouped by its inferred context. +Conditions are executed by context group in hierarchical order (resource → scope → signal-specific). + +For example: + +```yaml +log_conditions: + - 'resource.attributes["host.name"] == "localhost"' # inferred as resource context + - 'scope.name == "my.scope"' # inferred as scope context + - 'log.body == "test"' # inferred as log context +``` + +### Advanced Config + +For more complex use cases you may need to use the Filter Processor's advanced configuration style to explicitly specify the context and apply additional settings to a group of conditions. + +Format: + +```yaml +filter: + error_mode: ignore + _conditions: + - context: string + error_mode: propagate + conditions: + - string + - string + - context: string + conditions: + - string +``` + +`context`: specifies the OTTL context for the conditions. See the [General Config](#general-config) table for valid values per signal type. + +`error_mode`: allows overriding the top-level `error_mode` for this specific group of conditions. See the [General Config](#general-config) table for valid values. + +`conditions`: a list of OTTL conditions. + +Example: + +```yaml +processors: + filter: + error_mode: propagate + trace_conditions: + - context: resource + conditions: + - 'attributes["host.name"] == "localhost"' + - context: span + error_mode: ignore + conditions: + - 'attributes["container.name"] == "app_container_1"' + - 'name == "app_3"' + - context: spanevent + conditions: + - 'attributes["grpc"] == true' + - 'IsMatch(name, ".*grpc.*")' + metric_conditions: + - context: metric + conditions: - 'name == "my.metric" and resource.attributes["my_label"] == "abc123"' - 'type == METRIC_DATA_TYPE_HISTOGRAM' - datapoint: + - context: datapoint + conditions: - 'metric.type == METRIC_DATA_TYPE_SUMMARY' - - 'resource.attributes["service.name"] == "my_service_name"' - logs: - log_record: - - 'IsMatch(body, ".*password.*")' - - 'severity_number < SEVERITY_NUMBER_WARN' - profiles: - profile: - - 'duration_unix_nano > 3000' + log_conditions: + - context: log + conditions: + - 'IsMatch(body, ".*password.*")' + - 'severity_number < SEVERITY_NUMBER_WARN' ``` +### Examples + #### Dropping data based on a resource attribute ```yaml processors: filter: error_mode: ignore - traces: - span: - - IsMatch(resource.attributes["k8s.pod.name"], "my-pod-name.*") + trace_conditions: + - 'resource.attributes["k8s.pod.name"] == "my-pod-name"' ``` #### Dropping metrics with invalid type @@ -101,9 +206,8 @@ processors: processors: filter: error_mode: ignore - metrics: - metric: - - type == METRIC_DATA_TYPE_NONE + metric_conditions: + - 'metric.type == METRIC_DATA_TYPE_NONE' ``` #### Dropping specific metric and value @@ -111,9 +215,8 @@ processors: processors: filter: error_mode: ignore - metrics: - datapoint: - - metric.name == "k8s.pod.phase" and value_int == 4 + metric_conditions: + - 'metric.name == "k8s.pod.phase" and datapoint.value_int == 4' ``` #### Dropping non-HTTP spans @@ -121,9 +224,8 @@ processors: processors: filter: error_mode: ignore - traces: - span: - - attributes["http.request.method"] == nil + trace_conditions: + - 'span.attributes["http.request.method"] == nil' ``` #### Dropping HTTP spans @@ -131,9 +233,8 @@ processors: processors: filter: error_mode: ignore - traces: - span: - - attributes["http.request.method"] != nil + trace_conditions: + - 'span.attributes["http.request.method"] != nil' ``` #### Dropping non-error spans with a duration of less than 1 second @@ -141,9 +242,8 @@ processors: processors: filter: error_mode: ignore - traces: - span: - - (end_time - start_time) < Duration("1s") and status.code != STATUS_CODE_ERROR + trace_conditions: + - '(span.end_time - span.start_time) < Duration("1s") and span.status.code != STATUS_CODE_ERROR' ``` ### OTTL Functions @@ -196,6 +296,23 @@ filter/keep_good_metrics: - 'HasAttrOnDatapoint("bad.metric", "true")' ``` +## Legacy Configuration + +The following configuration options are deprecated and will be removed in a future release. Please migrate to the new `*_conditions` format. + +| Deprecated Config | Migrate To | +|----------------------|---------------------------------------------| +| `traces.resource` | `trace_conditions` with `resource.` prefix | +| `traces.span` | `trace_conditions` with `span.` prefix | +| `traces.spanevent` | `trace_conditions` with `spanevent.` prefix | +| `metrics.resource` | `metric_conditions` with `resource.` prefix | +| `metrics.metric` | `metric_conditions` with `metric.` prefix | +| `metrics.datapoint` | `metric_conditions` with `datapoint.` prefix| +| `logs.resource` | `log_conditions` with `resource.` prefix | +| `logs.log_record` | `log_conditions` with `log.` prefix | +| `profiles.resource` | `profile_conditions` with `resource.` prefix| +| `profiles.profile` | `profile_conditions` with `profile.` prefix | + ## Troubleshooting When using OTTL you can enable debug logging in the collector to print out useful information, @@ -213,9 +330,8 @@ receivers: processors: filter: error_mode: ignore - logs: - log_record: - - body == "test" + log_conditions: + - 'log.body == "test"' exporters: debug: