OTLP/HTTP Response Failures: Clarification needed for encoding used for failures

[psx95]

Issue

The spec for OTLP/HTTP Response Failures states that the

The response body for all HTTP 4xx and HTTP 5xx responses MUST be a Protobuf-encoded Status message that describes the problem.

This contradicts the encoding text mentioned in OTLP/HTTP Response paragraph that suggests:

The server MUST use the same "Content-Type" in the response as it received in the request.

Is this just a mistake or is there a reason as to why the response body for all HTTP 4xx and 5xx responses must be protobuf-encoded?

Consequence

The OTLP http exporters used in the OpenTelemetry Collector seem to adhere to this spec and always attempt to parse the error responses (4xx - 5xx errors) using proto.Unmarshal without checking the content-type.
This parsing fails when the server sets the content type as application/json (IIUC, the exporters should use json.Unmarshal to parse content-type application/json).

[tigrannajaryan]

I think this is a mistake. I think we should treat it as a spec bug.

Assuming that we go ahead and fix the spec bug I think we should make the Collector exporter implementation handle this change gracefully, i.e. attempt to unmarshal based on Content-Type and if that fails fall back to the current behavior.

[psx95]

I can send a PR to fix this, please feel free to assign this issue to me.

[tigrannajaryan]

Let's wait for more comments from @open-telemetry/spec-sponsors and @open-telemetry/technical-committee

[quentinmit]

Why doesn't this use normal content negotiation with the Accept header? You could fall back to defaulting to match the request's Content-type, but we already have a mechanism in HTTP for clients to control the response type they will get.

[tigrannajaryan]

Why doesn't this use normal content negotiation with the Accept header?

Agree, that would be the right way to do it, but too late to change that now unless we see a way to make the change in a backward and forward compatible way.

[quentinmit]

Why doesn't this use normal content negotiation with the Accept header?

Agree, that would be the right way to do it, but too late to change that now unless we see a way to make the change in a backward and forward compatible way.

Why would it not be backwards compatible to start honoring Accept when present? I would assume that current clients don't send it, or if they do they would send it with something like */* or text/*. I don't think it would be a breaking change if you continued falling back to the existing behavior when you can't match the requested content types. (But isn't this issue about the fact that the current behavior is already broken?)