chore(tracing): track spans_dropped telemetry for the trace writer pipeline#18967
chore(tracing): track spans_dropped telemetry for the trace writer pipeline#18967mabdinur wants to merge 5 commits into
Conversation
…-trace-dotnet AgentlessTraceWriter previously had no instrumentation telemetry, making it hard to answer basic questions like how many spans were encoded, sent, or dropped and why. This adds spans_enqueued_for_serialization, trace_chunks_sent, trace_chunks_dropped (reason: overfull_buffer/serialization_error/api_error), and trace_api.requests/responses/errors metrics, matching the metric and tag naming used for the equivalent HTTP client telemetry in dd-trace-dotnet. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Codeowners resolved as |
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ea84ea7a60
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
BenchmarksBenchmark execution time: 2026-07-10 21:11:14 Comparing candidate commit a79c7d8 in PR branch Found 0 performance improvements and 2 performance regressions! Performance is the same for 384 metrics, 9 unstable metrics. scenario:span-start
scenario:telemetryaddmetric-1-count-metric-1-times
|
mabdinur
added
the
changelog/no-changelog
There was a problem hiding this comment.
Pull request overview
This PR adds instrumentation telemetry metrics for the shared HTTPWriter send pipeline so that agentless trace export (and any other writers using HTTPWriter) reports span/chunk enqueue, send attempts, drop reasons, and per-HTTP-attempt request/response/error counts—aligned with dd-trace-dotnet’s metric/tag naming.
Changes:
- Added tracer telemetry recording to
HTTPWriterfor enqueue, drop reasons, chunk-send attempts, and HTTP request/response/error events. - Introduced new telemetry helper functions under
ddtrace/_trace/telemetry.pyfor writer-related metrics. - Added unit tests covering the new telemetry behavior and error-path tagging.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
ddtrace/internal/writer/writer.py |
Records tracer telemetry metrics at key points in the HTTP writer enqueue/encode/send pipeline. |
ddtrace/_trace/telemetry.py |
Adds telemetry helper functions for writer enqueue/sent/dropped and trace API request/response/error counts. |
tests/tracer/test_writer.py |
Adds TestWriterTelemetry to validate telemetry calls for success and failure modes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
mabdinur
changed the title
Record trace-processor span drops via the internal telemetry writer that SpanAggregator already imports, instead of ddtrace._trace.telemetry. The _trace.processor -> _trace.telemetry edge introduced new import cycles detected by the detect_circular_imports CI job. Co-authored-by: Cursor <cursoragent@cursor.com>
Replace the specialized HTTPWriter _telemetry_* methods with a generic record_trace_writer_metric helper, always pass an int span count, and simplify the writer telemetry tests to a focused, non-exhaustive set. Co-authored-by: Cursor <cursoragent@cursor.com>
mabdinur
changed the title
Summary
AgentlessTraceWriter/AgentlessWriterClientpreviously reported no instrumentation telemetry, unlikeNativeWriterand the LLMObs writer. This makes it hard to answer the operational question that matters most: how many spans are we losing, and where?This PR adds span-level drop telemetry (
TELEMETRY_NAMESPACE.TRACERS) across the writer pipeline. We deliberately track spans, not trace chunks:spans_enqueued_for_serialization— spans accepted into the encoder buffer.spans_dropped(tagreason) — spans that are definitively dropped, recorded once we are sure the span will never reach the intake. Reasons map to the four pipeline stages:tp_drop— dropped by a trace processor in theSpanAggregator(filtering / whole-trace drop).overfull_buffer— the encoder buffer could not fit the trace (BufferFull).serialization_error— the trace could not be encoded/compressed (BufferItemTooLarge,NoEncodableSpansError,encode()failure, gzip failure).api_error— the payload could not be delivered to the intake.trace_api.requests/trace_api.responses(tagstatus_code) /trace_api.errors(tagtype:timeout/network/status_code) — per-HTTP-attempt telemetry, including retries.Retry-safety (only count once we are SURE)
spans_droppedis never recorded speculatively on an attempt that might still succeed:_send_payload_with_backoff,RETRY_ATTEMPTS). Theapi_errordrop is recorded only after all retries are exhausted, in_flush_single_payload.>= 400) is not retried (the backoff stops as soon as aResponseis returned), so it is recorded once, after that single definitive attempt._send_payloadonly emits per-attempttrace_api.*counters; it never records span drops.Threading span counts through the pipeline
The buffer/encode/HTTP layers only knew the trace (chunk) count.
AgentlessTraceJSONEncodernow tracks apending_spanscount so the writer can attribute a real span count to a dropped payload. Encoders that don't expose an integer span count (e.g. CIVisibility) reportNoneand are skipped for span-level drops —trace_api.*and the existing DogStatsD metrics are unchanged.Test plan
TestWriterTelemetryintests/tracer/test_writer.py: spans enqueued;spans_droppedforoverfull_buffer/serialization_error(buffer, encode, gzip) /api_error(HTTP status and retries-exhausted); no drop on success; no drop when span count is unknown;_send_payloadrecords onlytrace_api.*(network / timeout / status_code), never span drops.test_processors.py:tp_droprecorded for whole-trace and partial processor drops, and a 0-count no-op when nothing is dropped.scripts/run-tests --venv ed437ab -- -- tests/tracer/test_writer.py tests/tracer/test_processors.py tests/tracer/test_encoders.py→ 232 passed, 10 skipped. The only failures are two pre-existing@pytest.mark.subprocesstests that require a local trace agent (Network error: client error (Connect)), confirmed to fail identically onmain.scripts/lint fmt/typingclean on all touched files.other:category — internal telemetry addition, no user-facing API/behavior change).