Describe the bug
With otel_trace_context inject (or propagate), an internal subrequest that proxies to an upstream does not get a traceparent written onto its outgoing request. The upstream call ends up disconnected from the trace (a new root) instead of nested under the request that spawned it.
I hit this serving a static HTML shell where a per-request fragment is pulled in with an SSI include that proxies to a backend. The page request is traced fine, but the SSI subrequest's call to the backend has no traceparent.
The cause is the early return for internal requests in onRequestStart:
|
ngx_int_t onRequestStart(ngx_http_request_t* r) |
|
{ |
|
// don't let internal redirects to override sampling decision |
|
if (r->internal) { |
|
return NGX_DECLINED; |
|
} |
So this is never reached:
|
if (lcf->traceContext & Propagation::Inject) { |
|
rc = inject(r, ctx->current); |
|
} |
SSI include virtual subrequests are internal, so they hit this guard and inject never runs for them.
To reproduce
- Build/run nginx with
ngx_otel_module (e.g. the nginx:alpine-otel image).
- Configure a server with
otel_trace on; and otel_trace_context propagate; (or inject).
- Serve an HTML file with
ssi on; containing <!--# include virtual="/_frag" -->.
- Add an internal location that proxies out:
location = /_frag {
internal;
proxy_pass http://backend;
}
- Request the page with an inbound
traceparent.
- On the backend, the call from the SSI subrequest arrives with no
traceparent and starts a new trace, rather than continuing the page's trace.
Expected behavior
The subrequest's proxy_pass should carry a traceparent (a child span of the request) so the backend call joins the same trace as the page request.
Your environment
Deploying nginx:alpine-otel (currently resolving to https://github.com/nginx/docker-nginx/blob/00348041cc12284266751457dcfc10b15645476a/mainline/alpine-otel/Dockerfile) to K8S
Additional context
I understand the guard — the comment points at internal redirects (try_files and similar), where you don't want a re-entry to redo the sampling decision. But r->internal also catches genuine subrequests that proxy to an upstream, which is exactly where propagation is wanted. Would it be reasonable to distinguish subrequests from internal redirects here, so an internal subrequest still injects on its upstream call?
Note the embedded variables do resolve for the subrequest (the getters build/restore the context via ensureOtelCtx), so a working workaround is to set the header by hand in the subrequest's location:
location = /_frag {
internal;
proxy_pass http://backend;
proxy_set_header traceparent "00-$otel_trace_id-$otel_span_id-0$otel_parent_sampled";
}
This connects the backend call to the trace, which suggests the inject path itself is the only thing the internal-request guard is (over-)blocking.
Describe the bug
With
otel_trace_context inject(orpropagate), an internal subrequest that proxies to an upstream does not get atraceparentwritten onto its outgoing request. The upstream call ends up disconnected from the trace (a new root) instead of nested under the request that spawned it.I hit this serving a static HTML shell where a per-request fragment is pulled in with an SSI include that proxies to a backend. The page request is traced fine, but the SSI subrequest's call to the backend has no
traceparent.The cause is the early return for internal requests in
onRequestStart:nginx-otel/src/http_module.cpp
Lines 356 to 361 in e3b6c98
So this is never reached:
nginx-otel/src/http_module.cpp
Lines 388 to 390 in e3b6c98
SSI
include virtualsubrequests are internal, so they hit this guard andinjectnever runs for them.To reproduce
ngx_otel_module(e.g. thenginx:alpine-otelimage).otel_trace on;andotel_trace_context propagate;(orinject).ssi on;containing<!--# include virtual="/_frag" -->.traceparent.traceparentand starts a new trace, rather than continuing the page's trace.Expected behavior
The subrequest's
proxy_passshould carry atraceparent(a child span of the request) so the backend call joins the same trace as the page request.Your environment
Deploying
nginx:alpine-otel(currently resolving to https://github.com/nginx/docker-nginx/blob/00348041cc12284266751457dcfc10b15645476a/mainline/alpine-otel/Dockerfile) to K8SAdditional context
I understand the guard — the comment points at internal redirects (
try_filesand similar), where you don't want a re-entry to redo the sampling decision. Butr->internalalso catches genuine subrequests that proxy to an upstream, which is exactly where propagation is wanted. Would it be reasonable to distinguish subrequests from internal redirects here, so an internal subrequest still injects on its upstream call?Note the embedded variables do resolve for the subrequest (the getters build/restore the context via
ensureOtelCtx), so a working workaround is to set the header by hand in the subrequest's location:This connects the backend call to the trace, which suggests the inject path itself is the only thing the internal-request guard is (over-)blocking.