OTEP: Logger.Enabled

[pellared]

The purpose of this OTEP is to have an agreement that the SDK needs both:

• a "hook" for Logger.Enabled for customization and flexibility e.g. via extending LogRecordProcessor with Enabled method, Add Enabled to LogRecordProcessor #4363
• new LoggerConfig fields to conveniently address the most popular use cases, especially configuring the minimum severity level, Add min_severity to LoggerConfig #4364

The OTEP tries to elaborate how/why it should work well with (trace) sampling as well as Events API.

Fixes #4207

Side note: I can create issues for each "open question" for tracking purposes.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[cijothomas]

One thing I am not so sure about.. Do we expect OTel SDK to provide such control, mimicking what is often provided by the Logging libraries themselves? They often provide sophisticated mechanisms, and it may not be worth replicating everything in OTel...
We can explicitly state that our desire is to provide some control, but not necessarily matching/competing with existing logging libraries?

[pellared]

I do not expect the Logs API to be as user-friendly as the logging libraries popular for given language.

However, I think that the SDK (the logging backend) should be able handle different logging processing scenarios. E.g. sending verbose logs via user_events and warning logs via OTLP (probably I could come up with better examples). I heard that someone wanted to send non-event log records and event records to different destinations. Moreover, there is the following question in the Events Basics OTEP:

How to support routing logs from the Logs API to a language-specific logging library while simultaneously routing logs from the language-specific logging library to an OpenTelemetry Logging Exporter?

I see logging libraries as logs frontend and Logs SDK as logs backend. Logs API is the thing which bridges between the frontend and backend.

Side note: I think that even without this use case we would get the same design.

[pellared]

I moved it as the last use-case given that it may be seen as the least important. a4257ef

[pellared]

Let me know if the current way it is described looks good overall.

[pellared]

@cijothomas, PTAL

I want to also add that some users may want to directly use Logs API without using any bridged library. See: #4352

[pellared]

During the Logs SIG meeting we agreed that LogRecordFilterer should be the "main" proposal (it is more loosely coupled, processors are not "chained" by design) and move the existing that adds Enabled to LogRecordProcessor as an alternative. It is also something which most people are familiar with.

EDIT: After a demo of the OTel Go Logs design it looks like an acceptable solution, but the OTEP should call out explicitly the optimizations we have done. E.g. that we are not looping through processors when none of them implements Enabled and that we are only looping thrrough the processors which implement enabled.

// Enabled returns true if at least one Processor held by the LoggerProvider
// that created the logger will process param for the provided context and param.
//
// If it is not possible to definitively determine the param will be
// processed, true will be returned by default. A value of false will only be
// returned if it can be positively verified that no Processor will process.
func (l *logger) Enabled(ctx context.Context, param log.EnabledParameters) bool {
	// If there are more Processors than FilterProcessors we cannot be sure
	// that all Processors will drop the record. Therefore, return true.
	//
	// If all Processors are FilterProcessors, check if any is enabled.
	return len(l.provider.processors) > len(l.provider.fltrs) || anyEnabled(ctx, param, l.provider.fltrs)
}

I also plan creating a PR for #4364 as this does not seem controversial. This should simplify this OTEP (assuming the PR will be merged).

[MrAlias]

Ideally, this is configurable on a per-processor level, not just SDK wide, so you can have alternate destinations for different log severity.

[pellared]

I agree. I will add this use case. Also given later I already described:

MinimumSeverityLevelProcessor is for configuring log processing pipelines.
It is the only choice when one would like to set the minimum severity level
for a certain exporting pipeline.
For example, one batching processor would be exporting logs only above info level
via OTLP and a second simple processor would be exporting all logs to stdout.

[MrAlias]

These number references are confusing as the seem to be referencing something from another section.

[MrAlias]

This seems to be doing too much. Why not just focus on adding logger enabled like the title indicates?

[pellared]

Having two "layers" of setting up the severity filtering may be very confusing for the users.
The behavior of LoggerConfig.min_severity could be achieved also via a filter processor that takes into consideration the InstrumentationScope when filtering by severity level.

Most probably for languages like Java LoggerConfig.min_severity won't be helpful as the severity level is already set on the bridged logging libraries.

The important piece is to support cases like described here: https://github.com/open-telemetry/opentelemetry-specification/pull/4290/files#r1925688450 which is not possible via LoggerConfig.min_severity

@jack-berg, @trask, any opinion?

@jack-berg, are all OTel SDKs are supposed to support LoggerConfig (assuming it gets stabilized) or is it is assumed as an optional feature? Couldn't we provide processors which result in the same behavior?

CC @MrAlias

[pellared]

psuedo-code of configuration that could be possible when we use processors instead of logger config

logs:
  processors:
    - min_sev_processor:
        level: info
        processor:
          batch_processor:
            exporter: otlp_log_exporter
    - simple_processor:
        exporter: stdoutlog_exporter

[pellared]

Side note: I think that LoggerConfig functionality could be replaced with a LogRecordProcessor decorator that implements the same thing. I may try to prototype it.

[pellared]

I think that LoggerConfig functionality could be replaced with a LogRecordProcessor decorator that implements the same thing. I may try to prototype it.

Here is a PR for OTel Go that improves the filtering processor so that it has instrumentation scope and resource in its input.

• [reflect] sdk/log/xlog: Add FilterProcessor and EnabledParameters opentelemetry-go#6286

Therefore, it can replace the the need of LoggerConfig.min_severity as one can make a processor that just filters the log records by severity and instrumentation scope.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[pellared]

I moving this OTEP to draft per this #4290 (comment). I start leaning towards just working on #4363 and maybe some clarifying spec improvements.

[cijothomas]

3. Configure a minimum log severity level on the SDK level.

Sharing my experience from OTel .NET, Rust. Such a capability is already provided by the existing Logging library.
For example, the following one-liner is enough to tell .NET's ILogger to only send INFO and above to OpenTelemetry.

builder.AddFilter<OpenTelemetryLoggerProvider>("*", LogLevel.INFO);

Similar in Rust. The following tells the logging library to send only Info and above to OTel

let filter_otel = EnvFilter::new("info");
let otel_layer = otel_layer.with_filter(filter_otel);

Can you share scenarios where this functionality had to be provided by OTel itself? Is this only for scenarios where users use OTel Logs/Event API and they need to achieve filtering?

[pellared]

Is this only for scenarios where users use OTel Logs/Event API and they need to achieve filtering?

This is one of the reasons. Thanks to it we get rid the overhead that would caused by bridge. Making complex OTel Events could cause significant overhead because the conversions done in the brigde have to rely on reflection. In Go, it would be better if instrumentation libraries use OTel Log's API directly. Partialy related: https://github.com/open-telemetry/opentelemetry-go/blob/main/log/DESIGN.md#reuse-slog

For OTel Semantical events and log records pecific processing/filtering it would be easier for the users to use the OTel Data Model.

We could provide filtering and processors that are related to OTel domain. E.g. filtering by span context. If we do not have this in via SDK then we would need to create such filter for each logging library.

Finally, I guess we would like to have some support for filtering in OTel Configuration files in the future.

[cijothomas]

is the user here referring to end-users? Are you expecting users to do this pattern?

do_something_important()
{
logger.log("I did something important", results = {results()});
}

do_something_important()
{
if(Span::GetActiveSpan().IsRecording())
{
logger.log("I did something important", results = {results()});
}
else
{
//don't do logging.
}
}

I think this should be a SDK setting and instrumentations should not have to worry about this. They always call logger.log. Upto SDK to decide (based on config) if the log must be dropped as the active span is not recorded.

(Sorry If I misunderstood the section!)

[pellared]

Yes. It could be also implemented via a Processor which filters out not recorded spans.

[cijothomas]

I'd like to get more clarification on this.

For 5 (when exporting to OS kernel facilities like etw/user-events), this is primarily a perf saving mechanism. If OS says nobody is interested in your log/event, then the producer can short circuit. This has to be done very quickly and typically only has 1-2 top-level parameter like Severity/EventName/ScopeName.

6 is referring to scenarios where users want to do filtering, primarily to control volume of telemetry exported. This usually is done based on LogRecord (i.e body/attributes/name/severity etc.). I believe this is already doable by writing normal LogRecordProcessors, like the one shown in #4407

In other words, I see 5 requires LogProcessor.Enabled, but 6 does not require.

[pellared]

Thanks. My description of 6 is indeed not clear. I meant that the perf saving mechanism works when the procesors are composed - supports the advanced scenarios which I described in #4407

[pellared]

@cijothomas, I just want to call out that I really appreciate your comments even though this PR is in draft state and not updated. It helps me preparing #4363, which I guess you are also in favor of.

For now I am closing this PR. I will focus on #4407. I will create an OTEP for it only if it would occur to be necessary. However, I think that the discussions we already had show that there is a need for adding an opt-in Enabled for the processor.

Among Go SIG we were evaluating a few times an alternative to provide some new "filter" abstraction which is decoupled from the "processor". However, we faced more issues than benefits going this route (some if this is described here, but there were more issues: open-telemetry/opentelemetry-go#5825 (comment) . With the current opt-in Processor.Enabled we faced less issues so far.
We also do not want to replicate all features from the logging libraries. If someone prefer the log4j (or other) filter design then someone can always use a bridge and use log4j for filtering. Enabled callback hook is the simplest design (yet very flexible) which makes it easy to implement in the SDKs. This design is inspired from the design of the two most popular Go structured logging libraries: https://pkg.go.dev/log/slog (standard library) and https://pkg.go.dev/go.uber.org/zap.

It is worth to adding that Rust design is similar and it also has an Enabled hook. See #4363 (comment). Basically we want to add something like https://docs.rs/log/latest/log/trait.Log.html#tymethod.enabled to the LogRecordProcessor and allow users to implement Enabled in the way that it will meet their requirements.

I also want to call out form https://github.com/open-telemetry/opentelemetry-specification/blob/main/oteps/0265-event-vision.md#open-questions:

How to support routing logs from the Logs API to a language-specific logging library

To support this we would need a log record processor which bridges the Logs API calls to given logging library. For such case we would need an Enabled hook in Processor to efficiently bridge Logger.Enabled calls. A filterer design would not satisfy such use case.

[cijothomas]

We also do not want to replicate all features from the logging libraries. If someone prefer the log4j (or other) filter design then someone can always use a bridge and use log4j for filtering

Full agree with this. (I also shared this opinion in offline discussion as well)