Summary
When exporting telemetry over gRPC using the OpenTelemetry Protocol (OTLP), the exporter may parse a server-provided grpc-status-details-bin trailer during retry handling. Prior to the fix, a malformed trailer could encode an extremely large length-delimited protobuf field which was used directly for allocation, allowing excessive memory allocation and potential denial of service (DoS).
Details
#5980 introduced a retry path that parses grpc-status-details-bin to extract gRPC retry delay information for retryable responses.
On that path:
OtlpGrpcExportClient captures grpc-status-details-bin from retryable status responses (ResourceExhausted / Unavailable).OtlpRetry invokes GrpcStatusDeserializer.TryGetGrpcRetryDelay using this untrusted trailer value.GrpcStatusDeserializer.DecodeBytes decoded a protobuf varint length and allocated new byte[length] without validating the bounds against the remaining payload size.
A malicious or compromised collector (or a MitM in weakly-protected deployments) could return a crafted grpc-status-details-bin payload that forces oversized allocation and memory exhaustion in the instrumented process.
Impact
If an OTLP/gRPC endpoint is attacker-controlled (or traffic is intercepted), a crafted retryable response can trigger large allocations during trailer parsing, which may exhaust memory and cause process instability/crash (availability impact / DoS).
Mitigation
The application's configured back-end/collector endpoint needs to behave maliciously. If the collector/back-end is a well-behaved implementation response bodies should not be excessively large if a request error occurs.
Workarounds
None known.
Remediation
#7064 updates GrpcStatusDeserializer to validate decoded length-delimited field sizes before allocation by ensuring the requested length is sane and does not exceed the remaining payload.
This causes malformed or truncated grpc-status-details-bin payloads to fail safely instead of attempting unbounded allocation.
References
Kielek Remediation developer
martincostello Remediation reviewer
arminru Coordinator
Summary
When exporting telemetry over gRPC using the OpenTelemetry Protocol (OTLP), the exporter may parse a server-provided
grpc-status-details-bintrailer during retry handling. Prior to the fix, a malformed trailer could encode an extremely large length-delimited protobuf field which was used directly for allocation, allowing excessive memory allocation and potential denial of service (DoS).Details
#5980 introduced a retry path that parses
grpc-status-details-binto extract gRPC retry delay information for retryable responses.On that path:
OtlpGrpcExportClientcapturesgrpc-status-details-binfrom retryable status responses (ResourceExhausted/Unavailable).OtlpRetryinvokesGrpcStatusDeserializer.TryGetGrpcRetryDelayusing this untrusted trailer value.GrpcStatusDeserializer.DecodeBytesdecoded a protobuf varint length and allocatednew byte[length]without validating the bounds against the remaining payload size.A malicious or compromised collector (or a MitM in weakly-protected deployments) could return a crafted
grpc-status-details-binpayload that forces oversized allocation and memory exhaustion in the instrumented process.Impact
If an OTLP/gRPC endpoint is attacker-controlled (or traffic is intercepted), a crafted retryable response can trigger large allocations during trailer parsing, which may exhaust memory and cause process instability/crash (availability impact / DoS).
Mitigation
The application's configured back-end/collector endpoint needs to behave maliciously. If the collector/back-end is a well-behaved implementation response bodies should not be excessively large if a request error occurs.
Workarounds
None known.
Remediation
#7064 updates
GrpcStatusDeserializerto validate decoded length-delimited field sizes before allocation by ensuring the requested length is sane and does not exceed the remaining payload.This causes malformed or truncated
grpc-status-details-binpayloads to fail safely instead of attempting unbounded allocation.References