Requirement
As a Jaeger user troubleshooting a trace that crosses an API gateway, I need guidance on using gateway telemetry as a diagnostic boundary.
The guidance should help determine whether a missing downstream span means:
the request never reached the downstream service;
trace context was lost between the gateway and the application;
application instrumentation or sampling prevented the span from being recorded; or
the problem occurred later in the telemetry pipeline.
This would help users verify the request path and trace-context propagation before investigating Jaeger ingestion or storage.
Problem
The current troubleshooting documentation covers sampling, application instrumentation, collectors, networking, logs, metrics, and missing spans in service-mesh environments. However, it does not provide a clear workflow for requests that pass through an API gateway.
A gateway can serve as a useful diagnostic boundary. When the gateway produces a span, users can compare the HTTP outcome, trace ID, parent relationship, and downstream spans to identify where trace continuity was lost.
I validated this approach in a version-pinned environment using Apache APISIX 3.17.0 with its OpenTelemetry plugin, OpenTelemetry Collector Contrib 0.157.0, and Jaeger 2.20.0.
The validation covered four scenarios across three clean Docker Compose starts on the same host:
A successful request produced APISIX and downstream service spans in the same trace.
An upstream-generated HTTP 500 still produced a connected service span, confirming that the request reached the application.
A gateway-side DNS failure returned HTTP 502 with no downstream service span because the request never reached the application.
A service that did not extract incoming trace context produced a separate root trace even though the HTTP request completed successfully.
These cases show that gateway telemetry can help distinguish request-path failures from trace-context propagation failures before users investigate Jaeger ingestion or storage.
Reproduction:
Yilialinn/apisix-jaeger-trace-troubleshooting-demo@ab33b8f
This is a contributor-maintained reproduction, not an official Apache APISIX or Jaeger release artifact.
Proposal
Add a short subsection to the Jaeger troubleshooting documentation describing how to validate missing or disconnected downstream spans at the gateway boundary.
A suggested investigation order is:
Inspect the gateway span and HTTP outcome.
Determine whether the response came from the downstream service or was generated by the gateway before an upstream connection was established.
If gateway and service spans both exist, compare their trace IDs and parent relationships.
Verify that W3C Trace Context is injected or forwarded by the gateway and extracted by the downstream application.
If the request reached the service but no span was recorded, check application instrumentation and sampling.
Investigate Collector export, Jaeger ingestion, and storage after confirming that the expected spans and trace context were produced.
The workflow should distinguish between gateway-generated failures, upstream-generated HTTP errors, and disconnected traces.
The guidance can remain gateway-agnostic. Apache APISIX could be mentioned as a validated example because its OpenTelemetry plugin produces gateway spans and exports them through an OpenTelemetry Collector.
The reproduction uses 100% sampling only to make controlled troubleshooting tests deterministic; this is not intended as a production recommendation.
If maintainers agree with this direction, I can follow up with a focused 250–400 word documentation PR against the page and version they recommend.
Open questions
- Is the existing operations troubleshooting page the preferred location, and should the initial change target development documentation only?
- Should Apache APISIX be mentioned as a validated example, or should the reproduction remain linked only from this issue?
- Should the subsection cover only the gateway-side workflow, or also include the handoff to Collector and Jaeger backend troubleshooting?
Requirement
As a Jaeger user troubleshooting a trace that crosses an API gateway, I need guidance on using gateway telemetry as a diagnostic boundary.
The guidance should help determine whether a missing downstream span means:
the request never reached the downstream service;
trace context was lost between the gateway and the application;
application instrumentation or sampling prevented the span from being recorded; or
the problem occurred later in the telemetry pipeline.
This would help users verify the request path and trace-context propagation before investigating Jaeger ingestion or storage.
Problem
The current troubleshooting documentation covers sampling, application instrumentation, collectors, networking, logs, metrics, and missing spans in service-mesh environments. However, it does not provide a clear workflow for requests that pass through an API gateway.
A gateway can serve as a useful diagnostic boundary. When the gateway produces a span, users can compare the HTTP outcome, trace ID, parent relationship, and downstream spans to identify where trace continuity was lost.
I validated this approach in a version-pinned environment using Apache APISIX 3.17.0 with its OpenTelemetry plugin, OpenTelemetry Collector Contrib 0.157.0, and Jaeger 2.20.0.
The validation covered four scenarios across three clean Docker Compose starts on the same host:
A successful request produced APISIX and downstream service spans in the same trace.
An upstream-generated HTTP 500 still produced a connected service span, confirming that the request reached the application.
A gateway-side DNS failure returned HTTP 502 with no downstream service span because the request never reached the application.
A service that did not extract incoming trace context produced a separate root trace even though the HTTP request completed successfully.
These cases show that gateway telemetry can help distinguish request-path failures from trace-context propagation failures before users investigate Jaeger ingestion or storage.
Reproduction:
Yilialinn/apisix-jaeger-trace-troubleshooting-demo@ab33b8f
This is a contributor-maintained reproduction, not an official Apache APISIX or Jaeger release artifact.
Proposal
Add a short subsection to the Jaeger troubleshooting documentation describing how to validate missing or disconnected downstream spans at the gateway boundary.
A suggested investigation order is:
Inspect the gateway span and HTTP outcome.
Determine whether the response came from the downstream service or was generated by the gateway before an upstream connection was established.
If gateway and service spans both exist, compare their trace IDs and parent relationships.
Verify that W3C Trace Context is injected or forwarded by the gateway and extracted by the downstream application.
If the request reached the service but no span was recorded, check application instrumentation and sampling.
Investigate Collector export, Jaeger ingestion, and storage after confirming that the expected spans and trace context were produced.
The workflow should distinguish between gateway-generated failures, upstream-generated HTTP errors, and disconnected traces.
The guidance can remain gateway-agnostic. Apache APISIX could be mentioned as a validated example because its OpenTelemetry plugin produces gateway spans and exports them through an OpenTelemetry Collector.
The reproduction uses 100% sampling only to make controlled troubleshooting tests deterministic; this is not intended as a production recommendation.
If maintainers agree with this direction, I can follow up with a focused 250–400 word documentation PR against the page and version they recommend.
Open questions