OTLP gRPC exporter fails with "code = Unavailable desc = " - logs not reaching aggregator #14044
Description
I have daemonset collector collecting losg and sending to aggregator collector. The OTLP exporter in daemonset collector repeatedly fails with an empty error description:
2025-10-20T12:59:49.661Z info internal/retry_sender.go:133 Exporting failed. Will retry the request after interval. {"resource": {"service.instance.id": "d6cb3225-2a4d-4db9-b004-7d70f90b228c", "service.name": "otelcol-k8s", "service.version": "0.134.0"}, "otelcol.component.id": "otlp", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "error": "rpc error: code = Unavailable desc = ", "interval": "25.805882163s"}
Daemonset collector code
config:
extensions:
health_check:
endpoint: 0.0.0.0:13133
oauth2client:
client_id_file: /etc/secrets/clientId
client_secret_file: /etc/secrets/clientSecret
token_url: https://example/token
tls:
insecure: false
timeout: 10s
endpoint_params:
audience: "observability-aggregator"
receivers:
filelog:
include:
- /var/log/pods/*/*/*.log
exclude:
- /var/log/pods/otel-system*/*/*.log
- /var/log/pods/observability-aggregator*/*/*.log
start_at: end
retry_on_failure:
enabled: true
include_file_path: true
include_file_name: false
operators:
- type: container
id: container-parser
max_log_size: 102400
kubeletstats:
collection_interval: 10s
auth_type: serviceAccount
endpoint: "https://${env:K8S_NODE_NAME}:10250"
insecure_skip_verify: true
metric_groups:
- node
- pod
- container
processors:
batch:
timeout: 1s
exporters:
debug:
verbosity: detailed
sampling_initial: 5
sampling_thereafter: 200
otlp:
endpoint: <grpc-route-endpoint>:443
auth:
authenticator: oauth2client
service:
extensions: [health_check, oauth2client]
pipelines:
logs:
receivers: [filelog]
processors: [batch]
exporters: [debug, otlp]
metrics:
receivers: [kubeletstats]
processors: [batch]
exporters: [debug] # , otlp
telemetry:
logs:
level: DEBUG
Aggregator collector config
extensions:
health_check:
endpoint: 0.0.0.0:13133
oidc:
providers:
- issuer_url: <example-endpoint>
audience: observability-aggregator
username_claim: preferred_username
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
auth:
authenticator: oidc
http:
endpoint: 0.0.0.0:4318
auth:
authenticator: oidc
processors:
resource:
attributes:
- key: auth.subject
action: upsert
from_context: auth.claims.sub
- key: auth.username
action: upsert
from_context: auth.claims.preferred_username
- key: auth.client_id
action: upsert
from_context: auth.claims.client_id
- key: tenant.source_system_name
action: upsert
from_context: auth.claims.example
connectors:
routing/logs:
default_pipelines: [logs/other]
error_mode: ignore
table:
- statement: route() where resource.attributes["tenant.source_system_name"] == "example"
pipelines: [logs/example]
exporters:
debug/example:
verbosity: normal
sampling_initial: 5
sampling_thereafter: 200
debug/other:
verbosity: normal
sampling_initial: 5
sampling_thereafter: 200
service:
extensions: [health_check, oidc]
pipelines:
logs/in:
receivers: [otlp]
processors: [resource]
exporters: [routing/logs]
logs/example:
receivers: [routing/logs]
processors: []
exporters: [debug/example]
logs/other:
receivers: [routing/logs]
processors: []
exporters: [debug/other]
telemetry:
logs:
level: DEBUG
Logs from aggregator collector
2025-10-20T12:57:00.899Z debug builders/builders.go:24 Alpha component. May change in the future. {"resource": {"service.instance.id": "1bc4c6ad-af18-4b67-a5f1-1d5ae8030d63", "service.name": "otelcol-contrib", "service.version": "0.137.0"}, "otelcol.component.id": "opensearch/toolchain", "otelcol.component.kind": "exporter", "otelcol.signal": "logs"}
2025-10-20T12:57:00.899Z debug builders/builders.go:24 Alpha component. May change in the future. {"resource": {"service.instance.id": "1bc4c6ad-af18-4b67-a5f1-1d5ae8030d63", "service.name": "otelcol-contrib", "service.version": "0.137.0"}, "otelcol.component.id": "opensearch/toolchain2", "otelcol.component.kind": "exporter", "otelcol.signal": "logs"}
2025-10-20T12:57:00.899Z debug builders/builders.go:24 Alpha component. May change in the future. {"resource": {"service.instance.id": "1bc4c6ad-af18-4b67-a5f1-1d5ae8030d63", "service.name": "otelcol-contrib", "service.version": "0.137.0"}, "otelcol.component.id": "debug/toolchain", "otelcol.component.kind": "exporter", "otelcol.signal": "logs"}
2025-10-20T12:57:00.900Z debug builders/builders.go:24 Alpha component. May change in the future. {"resource": {"service.instance.id": "1bc4c6ad-af18-4b67-a5f1-1d5ae8030d63", "service.name": "otelcol-contrib", "service.version": "0.137.0"}, "otelcol.component.id": "debug/other", "otelcol.component.kind": "exporter", "otelcol.signal": "logs"}
2025-10-20T12:57:00.900Z debug builders/builders.go:24 Alpha component. May change in the future. {"resource": {"service.instance.id": "1bc4c6ad-af18-4b67-a5f1-1d5ae8030d63", "service.name": "otelcol-contrib", "service.version": "0.137.0"}, "otelcol.component.id": "debug/app02", "otelcol.component.kind": "exporter", "otelcol.signal": "logs"}
2025-10-20T12:57:00.900Z debug builders/builders.go:24 Alpha component. May change in the future. {"resource": {"service.instance.id": "1bc4c6ad-af18-4b67-a5f1-1d5ae8030d63", "service.name": "otelcol-contrib", "service.version": "0.137.0"}, "otelcol.component.id": "routing/logs", "otelcol.component.kind": "connector", "otelcol.signal": "logs", "otelcol.signal.output": "logs"}
2025-10-20T12:57:00.901Z debug builders/builders.go:24 Beta component. May change in the future. {"resource": {"service.instance.id": "1bc4c6ad-af18-4b67-a5f1-1d5ae8030d63", "service.name": "otelcol-contrib", "service.version": "0.137.0"}, "otelcol.component.id": "resource", "otelcol.component.kind": "processor", "otelcol.pipeline.id": "logs/in", "otelcol.signal": "logs"}
2025-10-20T12:57:00.901Z debug builders/builders.go:24 Alpha component. May change in the future. {"resource": {"service.instance.id": "1bc4c6ad-af18-4b67-a5f1-1d5ae8030d63", "service.name": "otelcol-contrib", "service.version": "0.137.0"}, "otelcol.component.id": "debug/other-metrics", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics"}
2025-10-20T12:57:00.901Z debug builders/builders.go:24 Alpha component. May change in the future. {"resource": {"service.instance.id": "1bc4c6ad-af18-4b67-a5f1-1d5ae8030d63", "service.name": "otelcol-contrib", "service.version": "0.137.0"}, "otelcol.component.id": "debug/toolchain-metrics", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics"}
2025-10-20T12:57:00.901Z debug builders/builders.go:24 Alpha component. May change in the future. {"resource": {"service.instance.id": "1bc4c6ad-af18-4b67-a5f1-1d5ae8030d63", "service.name": "otelcol-contrib", "service.version": "0.137.0"}, "otelcol.component.id": "debug/app02-metrics", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics"}
2025-10-20T12:57:00.901Z debug builders/builders.go:24 Alpha component. May change in the future. {"resource": {"service.instance.id": "1bc4c6ad-af18-4b67-a5f1-1d5ae8030d63", "service.name": "otelcol-contrib", "service.version": "0.137.0"}, "otelcol.component.id": "routing/metrics", "otelcol.component.kind": "connector", "otelcol.signal": "metrics", "otelcol.signal.output": "metrics"}
2025-10-20T12:57:00.902Z debug builders/builders.go:24 Beta component. May change in the future. {"resource": {"service.instance.id": "1bc4c6ad-af18-4b67-a5f1-1d5ae8030d63", "service.name": "otelcol-contrib", "service.version": "0.137.0"}, "otelcol.component.id": "resource", "otelcol.component.kind": "processor", "otelcol.pipeline.id": "metrics/in", "otelcol.signal": "metrics"}
2025-10-20T12:57:00.902Z debug builders/builders.go:24 Stable component. {"resource": {"service.instance.id": "1bc4c6ad-af18-4b67-a5f1-1d5ae8030d63", "service.name": "otelcol-contrib", "service.version": "0.137.0"}, "otelcol.component.id": "otlp", "otelcol.component.kind": "receiver", "otelcol.signal": "metrics"}
2025-10-20T12:57:00.902Z debug [email protected]/otlp.go:58 created signal-agnostic logger {"resource": {"service.instance.id": "1bc4c6ad-af18-4b67-a5f1-1d5ae8030d63", "service.name": "otelcol-contrib", "service.version": "0.137.0"}, "otelcol.component.id": "otlp", "otelcol.component.kind": "receiver"}
2025-10-20T12:57:00.902Z debug builders/builders.go:24 Stable component. {"resource": {"service.instance.id": "1bc4c6ad-af18-4b67-a5f1-1d5ae8030d63", "service.name": "otelcol-contrib", "service.version": "0.137.0"}, "otelcol.component.id": "otlp", "otelcol.component.kind": "receiver", "otelcol.signal": "logs"}
2025-10-20T12:57:00.902Z info builders/extension.go:50 Unmaintained component. Actively looking for contributors. Component will become deprecated after 3 months of remaining unmaintained. {"resource": {"service.instance.id": "1bc4c6ad-af18-4b67-a5f1-1d5ae8030d63", "service.name": "otelcol-contrib", "service.version": "0.137.0"}, "otelcol.component.id": "health_check", "otelcol.component.kind": "extension"}
2025-10-20T12:57:00.902Z debug builders/extension.go:48 Beta component. May change in the future. {"resource": {"service.instance.id": "1bc4c6ad-af18-4b67-a5f1-1d5ae8030d63", "service.name": "otelcol-contrib", "service.version": "0.137.0"}, "otelcol.component.id": "oidc", "otelcol.component.kind": "extension"}
@v0.137.0/service.go:222 Starting otelcol-contrib... {"resource": {"service.instance.id": "1bc4c6ad-af18-4b67-a5f1-1d5ae8030d63", "service.name": "otelcol-contrib", "service.version": "0.137.0"}, "Version": "0.137.0", "NumCPU": 8}
2025-10-20T12:57:00.903Z info extensions/extensions.go:41 Starting extensions... {"resource": {"service.instance.id": "1bc4c6ad-af18-4b67-a5f1-1d5ae8030d63", "service.name": "otelcol-contrib", "service.version": "0.137.0"}}
2025-10-20T12:57:00.903Z info extensions/extensions.go:45 Extension is starting... {"resource": {"service.instance.id": "1bc4c6ad-af18-4b67-a5f1-1d5ae8030d63", "service.name": "otelcol-contrib", "service.version": "0.137.0"}, "otelcol.component.id": "oidc", "otelcol.component.kind": "extension"}
2025-10-20T12:57:00.929Z info extensions/extensions.go:62 Extension started. {"resource": {"service.instance.id": "1bc4c6ad-af18-4b67-a5f1-1d5ae8030d63", "service.name": "otelcol-contrib", "service.version": "0.137.0"}, "otelcol.component.id": "oidc", "otelcol.component.kind": "extension"}
2025-10-20T12:57:00.929Z info extensions/extensions.go:45 Extension is starting... {"resource": {"service.instance.id": "1bc4c6ad-af18-4b67-a5f1-1d5ae8030d63", "service.name": "otelcol-contrib", "service.version": "0.137.0"}, "otelcol.component.id": "health_check", "otelcol.component.kind": "extension"}
2025-10-20T12:57:00.929Z info [email protected]/healthcheckextension.go:32 Starting health_check extension {"resource": {"service.instance.id": "1bc4c6ad-af18-4b67-a5f1-1d5ae8030d63", "service.name": "otelcol-contrib", "service.version": "0.137.0"}, "otelcol.component.id": "health_check", "otelcol.component.kind": "extension", "config": {"Endpoint":"0.0.0.0:13133","TLS":{},"CORS":{},"Auth":{},"MaxRequestBodySize":0,"IncludeMetadata":false,"ResponseHeaders":null,"CompressionAlgorithms":null,"ReadTimeout":0,"ReadHeaderTimeout":0,"WriteTimeout":0,"IdleTimeout":0,"Middlewares":null,"Path":"/","ResponseBody":null,"CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}
2025-10-20T12:57:00.929Z info extensions/extensions.go:62 Extension started. {"resource": {"service.instance.id": "1bc4c6ad-af18-4b67-a5f1-1d5ae8030d63", "service.name": "otelcol-contrib", "service.version": "0.137.0"}, "otelcol.component.id": "health_check", "otelcol.component.kind": "extension"}
2025-10-20T12:57:00.930Z info [email protected]/server.go:715 [core] [Server #1]Server created {"resource": {"service.instance.id": "1bc4c6ad-af18-4b67-a5f1-1d5ae8030d63", "service.name": "otelcol-contrib", "service.version": "0.137.0"}, "grpc_log": true}
2025-10-20T12:57:00.930Z info [email protected]/otlp.go:121 Starting GRPC server {"resource": {"service.instance.id": "1bc4c6ad-af18-4b67-a5f1-1d5ae8030d63", "service.name": "otelcol-contrib", "service.version": "0.137.0"}, "otelcol.component.id": "otlp", "otelcol.component.kind": "receiver", "endpoint": "[::]:4317"}
2025-10-20T12:57:00.930Z info [email protected]/otlp.go:179 Starting HTTP server {"resource": {"service.instance.id": "1bc4c6ad-af18-4b67-a5f1-1d5ae8030d63", "service.name": "otelcol-contrib", "service.version": "0.137.0"}, "otelcol.component.id": "otlp", "otelcol.component.kind": "receiver", "endpoint": "[::]:4318"}
2025-10-20T12:57:00.930Z info healthcheck/handler.go:131 Health Check state change {"resource": {"service.instance.id": "1bc4c6ad-af18-4b67-a5f1-1d5ae8030d63", "service.name": "otelcol-contrib", "service.version": "0.137.0"}, "otelcol.component.id": "health_check", "otelcol.component.kind": "extension", "status": "ready"}
2025-10-20T12:57:00.930Z info [email protected]/server.go:911 [core] [Server #1 ListenSocket #2]ListenSocket created {"resource": {"service.instance.id": "1bc4c6ad-af18-4b67-a5f1-1d5ae8030d63", "service.name": "otelcol-contrib", "service.version": "0.137.0"}, "grpc_log": true}
2025-10-20T12:57:00.930Z info [email protected]/service.go:245 Everything is ready. Begin running and processing data. {"resource": {"service.instance.id": "1bc4c6ad-af18-4b67-a5f1-1d5ae8030d63", "service.name": "otelcol-contrib", "service.version": "0.137.0"}}
Can somebody point out if any issues with the configuration, or why potentially is the export failing with empty error message