Set Otel log level based on parent logger

[mbrandenburger]

This PR addresses the issue reported in #1010

• fix hardcoded debug log level
• add log level check for expensive logging parms, such as, debug.Stack or base64 encoding
• fix incorrect log caller for context based loggers

[adecaro]

Indeed, there was an issue there with the logging but now with this change to have the traces we need to set the logger level to debug. The side effect is that we will have lots of output on the terminal.

Wondering if we can have a way to set a different level for Otel alone.

[adecaro]

we could use the already existing funciton

func Init(c Config) {
flogging.Init(c)
}

to allow a further customization for the otel level. So that even if we have info for the terminal, we have debug for otel.

[mbrandenburger]

as discussed offline, to keep configuration simple, we use our logging level for normal logs and "trace" event/logs. Introducing yet another customization for having different log levels for the logger and tracer may make the system more complicated.

Generally, we should use the tracing system to "track" the lifecycle of requests through the system and learn the interaction between multiple components and their execution latency. Tracing events/logs should give additional information if necessary, to better understand certain component execution flows. For details debugging, the component log should be used.
That is, we should aim to not overuse trace events/logs within spans, to maintain good readability and reduce additional performance overhead induced by tracing.

[mbrandenburger]

as discussed offline, to keep configuration simple, we use our logging level for normal logs and "trace" event/logs. Introducing yet another customization for having different log levels for the logger and tracer may make the system more complicated.

Generally, we should use the tracing system to "track" the lifecycle of requests through the system and learn the interaction between multiple components and their execution latency. Tracing events/logs should give additional information if necessary, to better understand certain component execution flows. For details debugging, the component log should be used. That is, we should aim to not overuse trace events/logs within spans, to maintain good readability and reduce additional performance overhead induced by tracing.

If we later feel, we would need to manage different logger/tracer logging-levels, we can introduce that feature later.

[arner]

Agree with the not overusing bit. If I understand correctly, all logs via ..fContext go through this logger (+tracer)? Then this PR will at least make it a lot quieter (unless explicitly activated). Can we still add trace information to spans without also logging?

[mbrandenburger]

Can we still add trace information to spans without also logging?

yes - you just need an instance of a span and then you create an event.

Example:

somespan.AddEvent("Read response")

[mbrandenburger]

Indeed, there was an issue there with the logging but now with this change to have the traces we need to set the logger level to debug. The side effect is that we will have lots of output on the terminal.

I was thinking that the "side effect" is the expected behavior. If one runs in debug mode, you see more details as normal. That would include logs and traces. The current behavior, however, is that even if you turn logging off, debug traces are captured.

Wondering if we can have a way to set a different level for Otel alone.

We could have this as a new feature.

[arner]

LGTM!