No way to drop unwanted trace headers

[sriemer]

Describe the bug

Some tracing vendors use vendor-specific trace headers which take precedence over the W3C trace context headers. Those are propagated as they are and cause trace propagation issues at the upstream service because that one does not pick up updated trace headers.

To reproduce

Steps to reproduce the behavior:

1. Set up and run a demo application consisting of (client app, nginx, and server-app) that is using vendor-specific trace headers and nginx-otel in the middle in the observability vendor environment.
2. Check trace correlation on the vendor UI
3. tcpdump the demo application

Expected behavior

There is a configuration option for nginx-otel to drop unwanted headers from the outbound HTTP request.

Your environment

• Commit June 4: e3b6c98
• x86_64

Additional context

With OpenTracing this was way easier but the data structures inside of nginx-otel are too limited for this.
Setting those headers to be dropped to empty string is not enough. They need to be dropped completely.

[sriemer]

Tried to use more_clear_headers directive in the same NGINX location as otel_trace_context propagate; and proxy_pass http://backend;. The issue is that nginx-otel is running in an earlier NGINX phase than the more_clear_headers does.

[route443]

Hello @sriemer, and thank you for bringing this issue to our attention.
The issue is clear, but I wouldn't rush to call it a bug or add a dedicated option to the module for removing vendor-specific trace headers.
The otel module inserts traceparent/tracestate directly into the list of incoming headers (r->headers_in) during the REWRITE phase, and subsequently, the proxy module copies all other incoming headers as-is when constructing the request to the upstream. The module lacks a mechanism to remove an arbitrary header.
As I understand it, more_clear_headers clears response headers sent to the client (an output header filter), it has absolutely nothing to do with the request sent to the upstream => in this case, the problem lies not in the phase, but in the direction.

As a working workaround, I'd recommend using proxy_set_header directive:

proxy_set_header x-vendor-trace ""

In this case, nginx will completely remove this header from the request to the upstream.
Let me know if this worked for you. Thanks!

[sriemer]

@route443 Works. Thank you very much! ❤️ 🙏 Document this and this can be closed, I would say.