Skip to content

feat!: use trace_based exemplar filter by default #2112

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bdewater-thatch wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat!: use trace_based exemplar filter by default #2112

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions metrics_sdk/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,8 @@ At this time, you should be able to:
* Export metrics on a schedule using `PeriodicMetricReader` with any compatible push exporter (e.g. OTLP via `opentelemetry-exporter-otlp-metrics`)
* Attach exemplars to metric data points for trace correlation:
* `AlwaysOnExemplarFilter` — every measurement is eligible
* `AlwaysOffExemplarFilter` — no exemplars collected (default)
* `TraceBasedExemplarFilter` — only measurements inside a sampled trace
* `AlwaysOffExemplarFilter` — no exemplars collected
* `TraceBasedExemplarFilter` — only measurements inside a sampled trace (default)

We do not yet have support for:

Expand Down Expand Up @@ -278,7 +278,7 @@ OpenTelemetry.meter_provider.shutdown

Exemplars attach individual raw measurements — along with the trace context active at the time of the measurement — to an exported metric data point. This lets you jump from a metric spike directly to the trace that caused it.

By default exemplars are **disabled** (`AlwaysOffExemplarFilter`). Enable them via an environment variable or programmatically.
By default, exemplars use `TraceBasedExemplarFilter`, which records measurements only when they occur inside a sampled trace.

#### Enable via environment variable

Expand All @@ -289,7 +289,7 @@ export OTEL_METRICS_EXEMPLAR_FILTER=trace_based
# Eligible for every measurement regardless of trace context
export OTEL_METRICS_EXEMPLAR_FILTER=always_on

# Disabled (default)
# Disabled
export OTEL_METRICS_EXEMPLAR_FILTER=always_off
```

Expand Down
8 changes: 3 additions & 5 deletions metrics_sdk/lib/opentelemetry/sdk/metrics/meter_provider.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,19 +130,17 @@ def register_synchronous_instrument(instrument)
end
alias register_asynchronous_instrument register_synchronous_instrument

# spec: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/configuration/sdk-environment-variables.md#exemplar
# this is one way to turn on the exemplar (exemplar should be turned off by default)
def exemplar_filter_setup
case ENV['OTEL_METRICS_EXEMPLAR_FILTER']
when 'always_on'
@exemplar_filter = Exemplar::AlwaysOnExemplarFilter
when 'trace_based'
when nil, '', 'trace_based'
@exemplar_filter = Exemplar::TraceBasedExemplarFilter
when 'always_off'
@exemplar_filter = Exemplar::AlwaysOffExemplarFilter
else
OpenTelemetry.logger.warn("OTEL_METRICS_EXEMPLAR_FILTER #{ENV['OTEL_METRICS_EXEMPLAR_FILTER']} is not part of the provided exemplar filters. Exemplar is off.")
@exemplar_filter = Exemplar::AlwaysOffExemplarFilter
OpenTelemetry.logger.warn("OTEL_METRICS_EXEMPLAR_FILTER #{ENV['OTEL_METRICS_EXEMPLAR_FILTER']} is not part of the provided exemplar filters. Using trace_based.")
@exemplar_filter = Exemplar::TraceBasedExemplarFilter
end
end

Expand Down
63 changes: 51 additions & 12 deletions metrics_sdk/test/opentelemetry/sdk/metrics/meter_provider_test.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,8 +141,32 @@
describe 'exemplar' do
describe '#exemplar_filter_setup' do
it 'without OTEL_METRICS_EXEMPLAR_FILTER' do
meter_provider = OpenTelemetry::SDK::Metrics::MeterProvider.new
_(meter_provider.exemplar_filter).must_equal OpenTelemetry::SDK::Metrics::Exemplar::AlwaysOffExemplarFilter
OpenTelemetry::TestHelpers.with_test_logger do |log_stream|
OpenTelemetry::TestHelpers.with_env('OTEL_METRICS_EXEMPLAR_FILTER' => nil) do
meter_provider = OpenTelemetry::SDK::Metrics::MeterProvider.new
_(meter_provider.exemplar_filter).must_equal OpenTelemetry::SDK::Metrics::Exemplar::TraceBasedExemplarFilter
end

_(log_stream.string).wont_match(/OTEL_METRICS_EXEMPLAR_FILTER/)
end
end

it 'with OTEL_METRICS_EXEMPLAR_FILTER as empty string' do
OpenTelemetry::TestHelpers.with_test_logger do |log_stream|
OpenTelemetry::TestHelpers.with_env('OTEL_METRICS_EXEMPLAR_FILTER' => '') do
meter_provider = OpenTelemetry::SDK::Metrics::MeterProvider.new
_(meter_provider.exemplar_filter).must_equal OpenTelemetry::SDK::Metrics::Exemplar::TraceBasedExemplarFilter
end

_(log_stream.string).wont_match(/OTEL_METRICS_EXEMPLAR_FILTER/)
end
end

it 'with OTEL_METRICS_EXEMPLAR_FILTER as trace_based' do
OpenTelemetry::TestHelpers.with_env('OTEL_METRICS_EXEMPLAR_FILTER' => 'trace_based') do
meter_provider = OpenTelemetry::SDK::Metrics::MeterProvider.new
_(meter_provider.exemplar_filter).must_equal OpenTelemetry::SDK::Metrics::Exemplar::TraceBasedExemplarFilter
end
end

it 'with OTEL_METRICS_EXEMPLAR_FILTER as always_on' do
Expand All @@ -152,12 +176,23 @@
end
end

it 'with OTEL_METRICS_EXEMPLAR_FILTER as invalid option' do
OpenTelemetry::TestHelpers.with_env('OTEL_METRICS_EXEMPLAR_FILTER' => 'better_on') do
it 'with OTEL_METRICS_EXEMPLAR_FILTER as always_off' do
OpenTelemetry::TestHelpers.with_env('OTEL_METRICS_EXEMPLAR_FILTER' => 'always_off') do
meter_provider = OpenTelemetry::SDK::Metrics::MeterProvider.new
_(meter_provider.exemplar_filter).must_equal OpenTelemetry::SDK::Metrics::Exemplar::AlwaysOffExemplarFilter
end
end

it 'with OTEL_METRICS_EXEMPLAR_FILTER as invalid option' do
OpenTelemetry::TestHelpers.with_test_logger do |log_stream|
OpenTelemetry::TestHelpers.with_env('OTEL_METRICS_EXEMPLAR_FILTER' => 'better_on') do
meter_provider = OpenTelemetry::SDK::Metrics::MeterProvider.new
_(meter_provider.exemplar_filter).must_equal OpenTelemetry::SDK::Metrics::Exemplar::TraceBasedExemplarFilter
end

_(log_stream.string).must_match(/OTEL_METRICS_EXEMPLAR_FILTER better_on is not part of the provided exemplar filters. Using trace_based./)
end
end
end

describe '#disable_exemplar_filter' do
Expand All @@ -174,19 +209,23 @@

describe '#enable_exemplar_filter' do
it 'will turn it on with default exemplar filter' do
meter_provider = OpenTelemetry::SDK::Metrics::MeterProvider.new
_(meter_provider.exemplar_filter).must_equal OpenTelemetry::SDK::Metrics::Exemplar::AlwaysOffExemplarFilter
OpenTelemetry::TestHelpers.with_env('OTEL_METRICS_EXEMPLAR_FILTER' => 'always_off') do
meter_provider = OpenTelemetry::SDK::Metrics::MeterProvider.new
_(meter_provider.exemplar_filter).must_equal OpenTelemetry::SDK::Metrics::Exemplar::AlwaysOffExemplarFilter

meter_provider.enable_exemplar_filter
_(meter_provider.exemplar_filter).must_equal OpenTelemetry::SDK::Metrics::Exemplar::TraceBasedExemplarFilter
meter_provider.enable_exemplar_filter
_(meter_provider.exemplar_filter).must_equal OpenTelemetry::SDK::Metrics::Exemplar::TraceBasedExemplarFilter
end
end

it 'will turn it on with customized always on' do
meter_provider = OpenTelemetry::SDK::Metrics::MeterProvider.new
_(meter_provider.exemplar_filter).must_equal OpenTelemetry::SDK::Metrics::Exemplar::AlwaysOffExemplarFilter
OpenTelemetry::TestHelpers.with_env('OTEL_METRICS_EXEMPLAR_FILTER' => 'always_off') do
meter_provider = OpenTelemetry::SDK::Metrics::MeterProvider.new
_(meter_provider.exemplar_filter).must_equal OpenTelemetry::SDK::Metrics::Exemplar::AlwaysOffExemplarFilter

meter_provider.enable_exemplar_filter(exemplar_filter: OpenTelemetry::SDK::Metrics::Exemplar::AlwaysOnExemplarFilter)
_(meter_provider.exemplar_filter).must_equal OpenTelemetry::SDK::Metrics::Exemplar::AlwaysOnExemplarFilter
meter_provider.enable_exemplar_filter(exemplar_filter: OpenTelemetry::SDK::Metrics::Exemplar::AlwaysOnExemplarFilter)
_(meter_provider.exemplar_filter).must_equal OpenTelemetry::SDK::Metrics::Exemplar::AlwaysOnExemplarFilter
end
end
end
end
Expand Down
3 changes: 0 additions & 3 deletions metrics_sdk/test/test_helper.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,3 @@ def right_boundary(scale, index)

result
end

# Suppress warn-level logs about a missing OTEL_METRICS_EXEMPLAR_FILTER for exemplars
ENV['OTEL_METRICS_EXEMPLAR_FILTER'] = 'always_off'
Loading