-
Notifications
You must be signed in to change notification settings - Fork 998
Description
Is your feature request related to a problem? Please describe.
We've been experiencing failures when exporting data from our Node.js SDK to the OpenTelemetry Collector through gRPC. After Collector restarts or deployments, the connections become unhealthy in environments with load balancers or network configurations that terminate idle connections.
To address this issue, we've tuned the settings for gRPC keepalive and reconnections in our Go SDK, and we're seeing good results so far. However, there's no equivalent capability in the Node.js OTLP gRPC exporters, since the @grpc/grpc-js channel options aren't exposed.
I've found an issue that dates back to 2023, which was automatically closed by the stale bot: #4024.
Describe the solution you'd like
The GrpcExporterTransport in @opentelemetry/otlp-grpc-exporter-base currently hardcodes only compression and user-agent options:
opentelemetry-js/experimental/packages/otlp-grpc-exporter-base/src/grpc-exporter-transport.ts
Lines 145 to 156 in b616ae1
| this._client = new clientConstructor( | |
| this._parameters.address, | |
| this._parameters.credentials(), | |
| { | |
| 'grpc.default_compression_algorithm': toGrpcCompression( | |
| this._parameters.compression | |
| ), | |
| 'grpc.primary_user_agent': createUserAgent( | |
| this._parameters.userAgent | |
| ), | |
| } | |
| ); |
The ChannelOptions interface in @grpc/grpc-js not only support the settings we need, but some other useful settings as well:
grpc.keepalive_time_ms- interval between pingsgrpc.keepalive_timeout_ms- timeout waiting for ping acknowledgementgrpc.keepalive_permit_without_calls- send pings even without active RPCsgrpc.initial_reconnect_backoff_ms/grpc.max_reconnect_backoff_ms- reconnection timing
A possible solution would be adding a channelOptions parameter to OTLPGRPCExporterConfigNode:
| export interface OTLPGRPCExporterConfigNode extends OTLPExporterConfigBase { |
export interface OTLPGRPCExporterConfigNode extends OTLPExporterConfigBase {
credentials?: ChannelCredentials;
metadata?: Metadata;
compression?: CompressionAlgorithm;
userAgent?: string;
channelOptions?: Partial<ChannelOptions>;
}Example usage:
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';
const exporter = new OTLPTraceExporter({
url: 'otel-collector.example.com:4317',
channelOptions: {
'grpc.keepalive_time_ms': 50000,
'grpc.keepalive_timeout_ms': 60000,
'grpc.keepalive_permit_without_calls': 1,
'grpc.initial_reconnect_backoff_ms': 5000,
'grpc.max_reconnect_backoff_ms': 5000,
},
});Describe alternatives you've considered
- Switching to HTTP exporter, but it loses the gRPC benefits.
- Implementing a custom
GrpcExporterTransport, but requires tracking upstream changes.
Additional context
- Go SDK parity: The Go OTLP exporter exposes
WithDialOption()for full gRPC configuration control. Should Node.js have an equivalent capability? - Once this issue has been discussed and considering we have a valid proposal, I'm willing to submit a PR implementing this feature.
Tip: React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.