Warning message about gRPC timeout is logged too frequently

[donbourne]

Summary

When Liberty is unable to connect to the OpenTelemetry Collector the OTel sdk logs messages to indicate that failure.

[WARNING] Failed to export logs. Server responded with gRPC status code 2. Error message: timeout
[WARNING] Failed to export metrics. Server responded with gRPC status code 2. Error message: timeout
[WARNING] Failed to export metrics. Server responded with gRPC status code 2. Error message: timeout
[WARNING] Failed to export logs. Server responded with gRPC status code 2. Error message: timeout
[WARNING] Failed to export metrics. Server responded with gRPC status code 2. Error message: timeout

the message appears to be logged by the otel sdk within the Liberty mpTelemetry-2.0 feature from the HttpExporter
interestingly... it's using a ThrottlingLogger to log that. you can see in output...

[WARNING] Failed to export logs. Server responded with gRPC status code 2. Error message: timeout
[WARNING] Failed to export logs. Server responded with gRPC status code 2. Error message: timeout
[WARNING] Failed to export metrics. Server responded with gRPC status code 2. Error message: timeout
[WARNING] Failed to export logs. Server responded with gRPC status code 2. Error message: timeout
[WARNING] Too many log messages detected. Will only log once per minute from now on.
[WARNING] Failed to export logs. Server responded with gRPC status code 2. Error message: timeout
[WARNING] Failed to export metrics. Server responded with gRPC status code 2. Error message: timeout
[WARNING] Failed to export metrics. Server responded with gRPC status code 2. Error message: timeout
[WARNING] Failed to export logs. Server responded with gRPC status code 2. Error message: timeout
[WARNING] Failed to export metrics. Server responded with gRPC status code 2. Error message: timeout

The ThrottlingLogger ensures that it is only logged once a minute - but that still seems like a lot.

Expected Behavior

The message should appear once for each of logs/metrics.

[WARNING] Failed to export logs. Server responded with gRPC status code 2. Error message: timeout
[WARNING] Failed to export metrics. Server responded with gRPC status code 2. Error message: timeout

Current Behavior

The message appears once a minute for each of logs/metrics.

Possible Solution

a few options to block that message from appearing in the logs too often:

1. rebuild the opentelemetry-java sdk ourselves and change the ThrottlingLogger behaviour
2. put a PR into opentelemetry-java sdk to limit the logging to about once an hour

Steps to Reproduce

set up Liberty to use mpTelemetry with the default OTLP exporters for logs/metrics
do not set up an OTel Collector

[ellen-lau]

Opened an issue in the opentelemetry-java repo: https://github.com/open-telemetry/opentelemetry-java/issues