Component(s)
exporter/googlecloud
Describe the issue you're reporting
I have read through #31033 which talks about metrics, and #34957 which talks about traces. I want to confirm that the same issue exists regarding logs, and that there's no way to solve it with configuration.
My use case is that I want to collect system logs and export them to GCP logging. It works fine when the network is up, but as soon as the network goes down, all logs are dropped. It would however be very useful to have these logs to diagnose connectivity issues.
Basic config example:
receivers:
journald:
storage: file_storage
priority: info
start_at: beginning
exporters:
googlecloud:
log:
default_log_name: otel-systemd-journal
As I understand it, the retry functionality of OpenTelemetry has been disabled, and so none of the functionality provided by sending_queue helps. We are forced to rely on the retry logic of the Google client library, which I think suggests that any outage lasting longer than 60 seconds will cause permanent loss of data https://github.com/googleapis/google-cloud-go/blob/5bfee69e5e6b46c99fb04df2c7f6de560abe0655/logging/apiv2/logging_client.go#L84-L97
As an aside I see no evidence that retry in the client library actually works. The first image shows log messages captured by OpenTelemetry during a network outage. Note the large gap in entries.
The second shows log entries captured by our custom logging app (which I was hoping to retire). The first log message reporting dropped entries appears only 1.3 seconds after the first log entry which was not uploaded.
I very much hope I've missed something basic here, but please confirm.
Tip
React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.
Component(s)
exporter/googlecloud
Describe the issue you're reporting
I have read through #31033 which talks about metrics, and #34957 which talks about traces. I want to confirm that the same issue exists regarding logs, and that there's no way to solve it with configuration.
My use case is that I want to collect system logs and export them to GCP logging. It works fine when the network is up, but as soon as the network goes down, all logs are dropped. It would however be very useful to have these logs to diagnose connectivity issues.
Basic config example:
As I understand it, the retry functionality of OpenTelemetry has been disabled, and so none of the functionality provided by
sending_queuehelps. We are forced to rely on the retry logic of the Google client library, which I think suggests that any outage lasting longer than 60 seconds will cause permanent loss of data https://github.com/googleapis/google-cloud-go/blob/5bfee69e5e6b46c99fb04df2c7f6de560abe0655/logging/apiv2/logging_client.go#L84-L97As an aside I see no evidence that retry in the client library actually works. The first image shows log messages captured by OpenTelemetry during a network outage. Note the large gap in entries.
The second shows log entries captured by our custom logging app (which I was hoping to retire). The first log message reporting dropped entries appears only 1.3 seconds after the first log entry which was not uploaded.
I very much hope I've missed something basic here, but please confirm.
Tip
React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding
+1orme too, to help us triage it. Learn more here.