Bug: OTLP responses should match content-type of requests

[fionaliao]

What is the bug?

From the OTLP spec:

The server MUST set “Content-Type: application/x-protobuf” header if the response body is binary-encoded Protobuf payload. The server MUST set “Content-Type: application/json” if the response is JSON-encoded Protobuf payload. The server MUST use the same “Content-Type” in the response as it received in the request.

Currently the the response format does not change depending on request format.
No content-type and a blank response is returned for success, and Content-Type: application/octet-stream + binary-encoded proto for errors:

mimir/pkg/distributor/otel.go

Line 342 in ba300f5

w.Header().Set("Content-Type", "application/octet-stream")

JSON requests should have Content-Type: application/json set in responses, with the responses following the JSON protobuf encoding, while protobuf requests should have Content-Type: application/x-protobuf set in responses (for the protobuf error case, the response body is fine but the Content-Type should be set correctly)

How to reproduce it?

A request like

curl -i -XPOST -H 'Content-Type: application/json' <mimir endpoint> -d '{
  "resourceMetrics": [
    {
      "scopeMetrics": [
        {
          "metrics": [
            {
              "name": "test_metric",
              "unit": "s",
              "description": "",
              "gauge": {
                "dataPoints": [
                  {
                    "asInt": 1,
                    "timeUnixNano": 1741602284008000000,
                    "attributes": [
                      {
                        "key": "bar_label",
                        "value": {
                          "stringValue": "abc"
                        }
                      }
                    ]
                  }
                ]
              }
            }
          ]
        }
      ]
    }
  ]
}' 

returns a 200 success without a content-type. For a valid json response {} should be returned. Similar issues apply to the other response combinations (success/error + request format)

What did you think would happen?

A response with Content-Type: application/json and body {}

What was your environment?

Issue can be seen in the current code (main currently at commit 20548c3)

Any additional context to share?

No response

[aknuds1]

Thanks for reporting this :)

[aknuds1]

@fionaliao and I have agreed that we need to seek spec clarification on whether errors should be JSON encoded if the request has this Content-Type. The spec is currently ambiguous on this point.

After re-reading the spec, we came to the conclusion that it must be the case that also error responses should have the same Content-Type as the request.