-
Notifications
You must be signed in to change notification settings - Fork 1.8k
Description
Component(s)
exporter/otlp
What happened?
Describe the bug
Since opentelemetry-collector v0.141.0, the dns:/// scheme in gRPC endpoints is no longer respected. The sanitizedEndpoint() function in configgrpc strips the dns:/// prefix before passing it to grpc.DialContext(), causing the endpoint to be treated as passthrough instead of using the DNS resolver.
This breaks DNS-based load balancing (e.g., round_robin with multiple IPs resolved from a single DNS name).
The root cause is in config/configgrpc/configgrpc.go:
- Line 258-270:
sanitizedEndpoint()strips thedns:///prefix - Line 314: The stripped endpoint is passed to
grpc.DialContext(), which defaults to passthrough resolver
Steps to reproduce
- Configure an OTLP exporter with DNS-based endpoint and load balancing:
exporters: otlp: endpoint: dns:///my-service.example.com:4317 balancer_name: round_robin
- The DNS name resolves to multiple IPs
- Observe that traffic only goes to one IP (passthrough behavior) instead of being distributed across all resolved IPs (DNS resolver behavior)
Workaround:
Use dns:///dns:/// as a prefix:
endpoint: dns:///dns:///my-service.example.com:4317What did you expect to see?
The dns:/// scheme should be preserved and passed to gRPC, enabling the DNS resolver to resolve the hostname and provide all IPs to the load balancer for proper client-side load balancing.
What did you see instead?
The dns:/// prefix is stripped, causing gRPC to use the passthrough resolver. This results in connections going to only one IP, completely bypassing client-side load balancing.
Related PRs:
- Cause: [configgrpc,exporter/otlp] Move gRPC client endpoint validation to configgrpc #14221 (moved endpoint validation to configgrpc, which strips the dns:/// scheme)
- Fix: [chore] move from DialContext to NewClient #13663 (migrate from grpc.DialContext to grpc.NewClient) - This PR should resolve the issue as grpc.NewClient properly handles URI schemes. Please prioritize merging this PR.
Collector version
0.142.0
Environment information
Environment
OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")
OpenTelemetry Collector configuration
exporters:
otlp:
endpoint: dns:///my-service.example.com:4317
balancer_name: round_robinLog output
Additional context
The deprecated grpc.DialContext treats bare hostnames as passthrough targets. The modern grpc.NewClient API properly parses URI schemes. PR #13663 addresses this but is blocked on test fixes.
Tip
React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.