Skip to content

Trace context not injected for internal subrequests (SSI include -> proxy_pass) #119

Description

@SimenB

Describe the bug

With otel_trace_context inject (or propagate), an internal subrequest that proxies to an upstream does not get a traceparent written onto its outgoing request. The upstream call ends up disconnected from the trace (a new root) instead of nested under the request that spawned it.

I hit this serving a static HTML shell where a per-request fragment is pulled in with an SSI include that proxies to a backend. The page request is traced fine, but the SSI subrequest's call to the backend has no traceparent.

The cause is the early return for internal requests in onRequestStart:

ngx_int_t onRequestStart(ngx_http_request_t* r)
{
// don't let internal redirects to override sampling decision
if (r->internal) {
return NGX_DECLINED;
}

So this is never reached:

if (lcf->traceContext & Propagation::Inject) {
rc = inject(r, ctx->current);
}

SSI include virtual subrequests are internal, so they hit this guard and inject never runs for them.

To reproduce

  1. Build/run nginx with ngx_otel_module (e.g. the nginx:alpine-otel image).
  2. Configure a server with otel_trace on; and otel_trace_context propagate; (or inject).
  3. Serve an HTML file with ssi on; containing <!--# include virtual="/_frag" -->.
  4. Add an internal location that proxies out:
location = /_frag {
    internal;
    proxy_pass http://backend;
}
  1. Request the page with an inbound traceparent.
  2. On the backend, the call from the SSI subrequest arrives with no traceparent and starts a new trace, rather than continuing the page's trace.

Expected behavior

The subrequest's proxy_pass should carry a traceparent (a child span of the request) so the backend call joins the same trace as the page request.

Your environment

Deploying nginx:alpine-otel (currently resolving to https://github.com/nginx/docker-nginx/blob/00348041cc12284266751457dcfc10b15645476a/mainline/alpine-otel/Dockerfile) to K8S

Additional context

I understand the guard — the comment points at internal redirects (try_files and similar), where you don't want a re-entry to redo the sampling decision. But r->internal also catches genuine subrequests that proxy to an upstream, which is exactly where propagation is wanted. Would it be reasonable to distinguish subrequests from internal redirects here, so an internal subrequest still injects on its upstream call?

Note the embedded variables do resolve for the subrequest (the getters build/restore the context via ensureOtelCtx), so a working workaround is to set the header by hand in the subrequest's location:

location = /_frag {
    internal;
    proxy_pass http://backend;
    proxy_set_header traceparent "00-$otel_trace_id-$otel_span_id-0$otel_parent_sampled";
}

This connects the backend call to the trace, which suggests the inject path itself is the only thing the internal-request guard is (over-)blocking.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    Status
    In Progress

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions