Skip to content

feat(instrumentation-http): add ignoreIncomingPropagationHook config - #6901

Open
matthieusieben wants to merge 1 commit into
open-telemetry:mainfrom
matthieusieben:http-ignore-incoming-propagation
Open

feat(instrumentation-http): add ignoreIncomingPropagationHook config#6901
matthieusieben wants to merge 1 commit into
open-telemetry:mainfrom
matthieusieben:http-ignore-incoming-propagation

Conversation

@matthieusieben

@matthieusieben matthieusieben commented Jul 9, 2026

Copy link
Copy Markdown

Which problem is this PR solving?

instrumentation-http unconditionally extracts the propagation context (traceparent, tracestate, baggage, …) from every incoming request's headers. For servers exposed to the public internet, this means any unauthenticated client can choose the trace ID that the server span — and the entire downstream fan-out — is recorded under.

Concrete consequences we've observed in production:

  • Trace pollution / spoofing: a client sending a fabricated traceparent co-locates our server spans under a trace ID it controls. Anyone on the internet can group arbitrary request spans under one trace.
  • Broken/orphaned traces: instrumented external clients (e.g. dd-trace in 64-bit mode) send their own traceparent; the root span lives in the client's tracing backend, so our backend shows <root span not yet received> for these traces.
  • Backend limits: a client accidentally (or deliberately) reusing a single context across a request loop merges many independent fan-outs into one giant trace, which can exceed backend per-trace limits (e.g. Tempo's max_bytes_per_trace), causing the whole trace — including our own spans — to be silently dropped.

There is currently no way to keep tracing incoming requests while not honoring their propagation headers. OTEL_PROPAGATORS=none is not an option since it also disables outbound injection, breaking internal trace-joining, and ignoreIncomingRequestHook disables tracing of the request entirely.

Short description of the changes

Adds an ignoreIncomingPropagationHook config option to HttpInstrumentationConfig:

new HttpInstrumentation({
  // Only trust propagation headers from internal callers.
  ignoreIncomingPropagationHook: (request) =>
    !isTrustedAddress(request.socket.remoteAddress),
});
  • Called with each incoming http.IncomingMessage before span creation. When it returns true, ROOT_CONTEXT is used instead of propagation.extract(ROOT_CONTEXT, headers) — the request is still traced normally, but the server span starts a new trace with no remote parent.
  • Receiving the full request (not just headers) allows trust decisions based on the socket's remote IP address, TLS state, etc.
  • Errors thrown by the hook are caught (safeExecuteInTheMiddle, matching ignoreIncomingRequestHook) and extraction proceeds normally — fail-open to today's behavior.
  • Outbound injection is unaffected, so internal service-to-service propagation keeps working.

Includes unit tests (hook true / false / throwing), README docs, and a changelog entry.

Alternative considered

Rather than an instrumentation-level hook, the propagation API itself could pass the request (carrier + connection metadata) to TextMapPropagator.extract() instead of only the headers carrier. A propagator could then decide, per request, to return the incoming context unchanged (root context) instead of a context linked to the remote parent. That would make the trust boundary expressible once in a custom propagator and reusable across all server instrumentations (http, grpc, fastify, …), instead of requiring each instrumentation to grow its own hook. It is, however, a wider API change (carrier semantics of TextMapGetter), so this PR takes the narrower instrumentation-scoped approach; happy to pivot if maintainers prefer the propagator-level design.

This was made by specifically directing Claude what to do.

@matthieusieben
matthieusieben requested a review from a team as a code owner July 9, 2026 12:58
@linux-foundation-easycla

linux-foundation-easycla Bot commented Jul 9, 2026

Copy link
Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.

  • ✅ login: matthieusieben / name: Matthieu Sieben (a7dcaf4)

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown

Welcome, contributor! Thank you for your contribution to opentelemetry-js.

Important reminders:

@matthieusieben
matthieusieben force-pushed the http-ignore-incoming-propagation branch 2 times, most recently from a7dcaf4 to 5f4cba5 Compare July 9, 2026 13:16
…option

Signed-off-by: Matthieu Sieben <matthieu.sieben@gmail.com>
Assisted-By: Claude Fable 5 (GitHub Copilot)
@matthieusieben
matthieusieben force-pushed the http-ignore-incoming-propagation branch from 5f4cba5 to 26565f5 Compare July 9, 2026 13:52
@pichlermarc

Copy link
Copy Markdown
Member

Hi @matthieusieben - thanks for opening this PR for discussion. This is actually an issue that's mentioned in our docs: https://opentelemetry.io/docs/concepts/context-propagation/#security-best-practices

Q: why not strip these headers at at something like a reverse proxy so that these never end up in the app at all? I suppose the idea is to don't allow external callers to add trace context/baggage but to allow internal ones. This is the most common way I've seen this issue handled in production deployments.

@matthieusieben

matthieusieben commented Jul 9, 2026

Copy link
Copy Markdown
Author

That sounds like a possible solution but in our deployment (in which we do have a reverse proxy), we allow calls from both internal and public systems. To make matters worse, some internal systems are deployed on dynamically provisioned systems (IPs).

Since we do receive credentials from those systems (which are verified at the application level, not in the proxy), it can be useful to be able to fine tune filtering at the application level, based on those credentials and the same verification logic.

Also (and this is not the case for us), NodeJS can be run without a reverse proxy, so having the knob here seems like a decent solution for those instances.

@matthieusieben

Copy link
Copy Markdown
Author

For the sake of discussion, I've had Claude create another option that would allow me to achieve the same thing using a custom Propagator:

#6902

@opentelemetry-pr-dashboard

opentelemetry-pr-dashboard Bot commented Jul 18, 2026

Copy link
Copy Markdown

Pull request dashboard status

Waiting on reviewers · refreshed 2026-07-29 01:44 UTC

Review the latest changes.

Status above doesn't look right?
  • Just replied or pushed? Anything around or after the refresh time above may not be picked up yet — give it a few minutes.
  • Anything look wrong? Report it with what you expected; it helps us improve the dashboard.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants