[receiver/otlpreceiver] Option to omit detailed error response

[locmai]

Component(s)

receiver/otlp

Is your feature request related to a problem? Please describe.

In some environment setups, exposing detailed error information in server responses can present security and privacy risks. As we are exposing our opentelemetry-collector to some external vendors, we have this risk evaluated and the error responses include some detailed on the implementations.

Describe the solution you'd like

• Introduce a new configuration parameter (e.g., suppress_error_details) for the otlpreceiver.
• When this option is enabled, the otlpreceiver should respond to client errors with a generic error message, omitting stack traces, internal error codes, or other detailed information.
• The default behavior should remain unchanged to preserve backward compatibility.

receivers:
  otlp:
    protocols:
      http:
        suppress_error_details: true

Describe alternatives you've considered

We have considered to have a proxy and override over the responses but would prefer a built-in option would reduce the need for external components or workarounds.

Additional context

We simply

curl -i -X POST https://example.com/otlp/v1/metrics \
  -H "Content-Type: application/json" \
  -d '{
    "resourceMetrics": [
      {
        "resource": {
          "attributes": [
            { "key": "service.name", "value": { "stringValue": "example-service" } }
          ]
        },
        "scopeMetrics": [
          {
            "metrics": [
              {
                "name": "broken_histogram_metric",
                "description": "A histogram with wrong bucket type",
                "unit": "1",
                "histogram": {
                  "dataPoints": [
                    {
                      "attributes": [],
                      "startTimeUnixNano": "1690000000000000000",
                      "timeUnixNano": "1690000001000000000",
                      "count": 3,
                      "sum": 42.0,
                      "bucketCounts": [1, 2, 0],
                      "explicitBounds": ["a", "b"]
                    }
                  ],
                  "aggregationTemporality": 2
                }
              }
            ]
          }
        ]
      }
    ]
  }'

And got the response with the detailed message about the ReadUint64:

{"code":3,"message":"ReadUint64: strconv.ParseFloat: parsing \"a\": invalid syntax, error found in #10 byte of ...|nds\": [\"a\", \"b\"]\n   |..., bigger context ...|, 0],\n                      \"explicitBounds\": [\"a\", \"b\"]\n                    }\n                  ],\n|..."}% 

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[axw]

I agree that the collector should have something built in for this. A few thoughts:

Defaulting to omitting details is IMO best, but there should probably be a way for components to explicitly indicate that some details are non-sensitive. If you always scrub all details it can degrade usability significantly, e.g. if a user mis-configures auth in their client application, it may be acceptable to provide them some hints about the expected format(s) without compromising security.

When omitting details from the response, it might be desirable to log the error with full details - depending on how the collector is deployed. I think this should be configurable.

I'd personally like to see a solution that supports all receivers and protocols (including gRPC). As an alternative to the proposed configuration, maybe a processor could introduced? A processor could intercept and optionally log errors before removing sensitive details. For gRPC an interceptor could be used, but for HTTP it's not really practical to use middleware since each HTTP receiver has its own way of presenting errors.

[dmathieu]

I agree that something more dynamic like a processor would be a better fit here.
Depending on your configuration or setup, you may want to send more or less data.

When omitting details from the response, it might be desirable to log the error with full details - depending on how the collector is deployed. I think this should be configurable.

We should already have the error in the emitted spans.

[jack5341]

Adding this config to the receiver doesn’t make sense to me either. It should belong to the processor component.
Besides that, what exactly do you mean by "more dynamic," @dmathieu? Do you mean changing the configuration level so that a simple config affects all protocols?

[jack5341]

Instead of placing it under the receiver, placing it in the processor.filter would be much more efficient and easier to understand, like in the example I created.

processors:
  filter:
    error_mode: ignore
    suppress_error_details: true
    traces:
      span:
        - 'attributes["container.name"] == "app_container_1"'
        - 'resource.attributes["host.name"] == "localhost"'
        - 'name == "app_3"'

I haven’t had the chance to test the collector in a production environment yet, so I’m wondering, what if a user wants to suppress error details for logs but not for traces?

[dmathieu]

I mean that you may want to send some errors as-is, and hide the details for others.

The filter processor is definitely not the right place for this. But a new/custom one may be a good fit.

[jack5341]

I read the issue again and realized I initially misunderstood it. Personally, I don’t see this as a significant security concern. In most cases, the OpenTelemetry Collector is exposed only to trusted external vendors that are explicitly chosen as you said, not random users.

From my perspective, the real risk starts when telemetry data includes PII or other sensitive information that should not be transmitted in the first place. Unless the collector is publicly accessible or shared in a multi-tenant setup, detailed error messages seem relatively low-risk.

[axw]

or shared in a multi-tenant setup

This is a legitimate setup. In such a setup, clients of the collector should typically not be privy to details of the backend, and errors often leak details.

[locmai]

Unless the collector is publicly accessible or shared in a multi-tenant setup

We have both of these cases.

Instead of placing it under the receiver, placing it in the processor.filter would be much more efficient and easier to understand, like in the example I created.

processors:
filter:
error_mode: ignore
suppress_error_details: true
traces:
span:
- 'attributes["container.name"] == "app_container_1"'
- 'resource.attributes["host.name"] == "localhost"'
- 'name == "app_3"'
I haven’t had the chance to test the collector in a production environment yet, so I’m wondering, what if a user wants to suppress error details for logs but not for traces?

We have tried this and the error was not suppressed. The reason we set this for receiver/otlp as we believe the errors were returned from the receiver (might be wrong here) from the data validation before even reaching to the processors. This is considered low risk on our side, but it would be great to have a path forward.

[axw]

The reason we set this for receiver/otlp as we believe the errors were returned from the receiver (might be wrong here) from the data validation before even reaching to the processors.

@locmai can you elaborate on the kinds of errors you're referring to here? Is this for example an error decoding the request payload? If so, what information leakage is there?

[locmai]

Thanks for following up, @axw!

Yes, this is specifically about errors during request payload validation - before the data reaches processors. In our case, it's the parsing/validation of OTLP metrics data, particularly when the collector attempts to parse fields like explicitBounds in histogram metrics.

Here's one concrete example from our security testing team:
When sending a malformed OTLP metrics payload with an XSS payload in the explicitBounds field:

"explicitBounds": [3, 600, 1200, 1800, 2200, "'\"--></style></scRipt><scRipt>netsparker(0x00959B)</scRipt>",
  3000, 3500, 4000, 4500]

The error response reflects back the malicious input unescaped:

{
    "code": 3,
    "message": "ReadUint64: strconv.ParseFloat: parsing 
  \"'\\\"--></style></scRipt><scRipt>netsparker(0x00959B)</scRipt>\": invalid syntax, error found in #10 byte of
   ...|</scRipt>\",3000,3500|..., bigger context ...|le></scRipt><scRipt>netsparker(0x00959B)</scRipt>\",3000,35
  00,4000,4500],\"count\":1,\"sum\":6248.3999999|..."
  }

Information Leakage Concerns:

1. Reflected XSS Vector: While the Content-Type is application/json (which reduces exploitability), the
   untrusted input is reflected without sanitization. In certain browser contexts or if the response is
   rendered/logged by client applications, this could be exploited.
2. Implementation Details: The error messages expose internal parsing logic (strconv.ParseFloat, byte
   positions, context snippets) which shouldn't be visible to external clients.
3. Multi-tenant Context: In our case, we're exposing OTLP endpoints to external vendors. Even if the XSS isn't
   directly exploitable, reflecting arbitrary attacker-controlled input in error responses violates security
   best practices and fails most security scanners.

[locmai]

Another option is to update these lines:

opentelemetry-collector/pdata/internal/json/iterator.go

Lines 71 to 75 in 0c5a4ff

	val, err := strconv.ParseInt(iter.ReadString(), 10, 64)
	if err != nil {
	iter.ReportError("ReadInt64", err.Error())
	return 0
	}

to just return a generic message