[BUG]: _dd.p.dm values not in the local SamplingMechanism enum are dropped

[jakub-bacic]

Tracer Version(s)

4.12.0 (verified), 3.19.8 (verified). The behaviour is present in every release since 2.21/3.11 (#13554).

Python Version(s)

Python 3.13.7 (not version specific — the code path is pure Python and independent of the interpreter version)

Pip Version(s)

pip 25.2

Bug Report

Summary

validate_sampling_decision() validates the incoming _dd.p.dm (decision maker) propagation tag against an allowlist built from the local SamplingMechanism enum (ddtrace/internal/sampling.py):

SAMPLING_MECHANISM_CONSTANTS = {
    "-{}".format(value) for name, value in vars(SamplingMechanism).items() if name.isupper()
}
...
if value not in SAMPLING_MECHANISM_CONSTANTS:
    del meta[SAMPLING_DECISION_TRACE_TAG_KEY]
    meta["_dd.propagation_error"] = "decoding_error"
    log.warning("failed to decode _dd.p.dm: %r", value)

Any sampling mechanism id the installed version doesn't know about is therefore treated as corrupt, even when the value is perfectly well-formed.

In production this is triggered by Datadog Synthetic API tests, which send x-datadog-tags: _dd.p.dm=-15. Mechanism 15 is not part of the shared mechanism registry, so every request from a Synthetic test produces a warning and loses its decision maker. As the receiving service, there is nothing I can configure to stop the sender.

What I observed:

• The warnings began at ~13:30 UTC on 2026-07-27, simultaneously across all of our Python-based microservices. Nothing was deployed on our side at that time, and no tracer version changed — so whatever started emitting -15 changed outside our control.
• I traced the affected requests back to a Datadog Synthetic API test, and confirmed the sender by inspecting that test's outgoing request headers directly: the request carries x-datadog-tags: _dd.p.dm=-15.

I can't tell why -15 started appearing on that date — that part is presumably a Datadog-side change and I have no visibility into it. Either way it's independent of the bug being reported here: the tracer should forward a well-formed mechanism id regardless of which component emits it, and regardless of when it was registered.

What happens

For each affected request:

1. _dd.p.dm is deleted from the extracted context, so the decision maker is lost for the whole local segment. Because the inbound sampling priority is already set, TraceSamplingProcessor skips the sampler, so no replacement _dd.p.dm is generated — the ingestion reason is gone for good, and the tag is not propagated further downstream either.
2. _dd.propagation_error: decoding_error is added to the context and ends up as a tag on the chunk-root span.
3. failed to decode _dd.p.dm: '-15' is logged at WARNING (rate limited to one per 60s by default, unbounded with DD_TRACE_LOGGING_RATE=0 or debug logging).

Sampling/retention itself is unaffected — the priority travels in its own header.

Why I think this is a bug, not intended strictness

1. The spec says the mechanism is an opaque integer. dd-trace-cpp states the rule explicitly in
include/datadog/sampling_mechanism.h:

Some tracer implementations do not decode SamplingPriority integer values outside of those
enumerated in this library. This makes adding new values infeasible, as older versions of tracers
propagating the SamplingPriority along the trace will omit new integer values. […] To allow
forward compatibility with future SamplingMechanism values, sampling mechanism is treated as
just an integer when being deserialized or serialized.

2. dd-trace-py is the only tracer that validates the mechanism value. I checked the extraction path in each implementation:

Tracer	Validation on extract	-15
Python	allowlist of locally known mechanism ids	dropped + warning + _dd.propagation_error
Java	syntax only — [10 hex service hash] "-" digit{digit} (ptags/PTagsCodec.java, validateDecisionMakerTag)	passes through
Node	generic key/value charset (text_map.js, _extractTags); tracestate accepts any int	passes through
Go	generic tagset parse + size limit (textmap.go, unmarshalPropagatingTagsIntoTrace)	passes through
.NET	charset only (Tagging/TagPropagation.cs, IsValid)	passes through
Ruby	generic codec decode (distributed/datadog.rb, extract_tags)	passes through
PHP	none — only resets dm when absent/conflicting priority (distributed_tracing_headers.c)	passes through
C++	does not inspect dm on extract	passes through
Rust / libdatadog	accepts any -N, N <= 255 (libdd-sampling, SamplingMechanism::from_str)	passes through

Java is the only other tracer that rejects anything here, and it rejects malformed syntax (-, --1, -1.0), not unknown ids. I also couldn't find any system-tests assertion requiring value-level validation of _dd.p.dm.

3. dd-trace-py itself is inconsistent about it. The same value on the same trace is accepted or rejected depending on which header carried it — the W3C tracestate extraction path applies no validation at all (see repro below). Span._set_sampling_decision_maker() likewise writes "-%d" % mechanism for any integer, and injection re-serializes whatever is in context._meta.

4. It couples the library to a registry it doesn't own. Any component that adopts a newly registered mechanism breaks decision-maker propagation for every dd-trace-py service downstream until those services upgrade. This is not hypothetical: on 3.19.x, -13 (AI Guard) is rejected for exactly the same reason, and #13516 was the same bug for -11/-12.

How it got here (for context)

• feat: add sampling decision to datadog trace tags #3797 introduced TRACE_TAG_RE = re.compile(r"^-([0-9])$"), a faithful transcription of the eBNF as written at the time (sampling mechanism = digit;) — correct only while the enum stopped at 7.
• The registry grew past 9, the single-digit regex started rejecting valid values → [BUG]: Warning "failed to decode _dd.p.dm: '-11'" logged with remote sampling configured #13516
• fix(sampling): validate_sampling_decision giving log warns on supported mechanisms #13554 fixed that by replacing the regex with the enum-derived allowlist. That silently changed the check from "is this well-formed?" to "is this a value I know?", which is what produces the current failure mode.

Proposed fix

Validate syntax and range instead of membership: accept -N where N is a decimal integer in 0..255 (matching libdatadog's u8 encoding), and keep rejecting genuinely malformed values (-, --1, -1.0, out-of-range) with the existing decoding_error behaviour. Unknown but well-formed mechanisms are then forwarded unchanged, and no future mechanism id requires a dd-trace-py release.

I have a patch and tests for this and will open a PR right after filing this issue.

Reproduction Code

# pip install ddtrace==4.12.0
import logging

logging.basicConfig(level=logging.WARNING, format="%(levelname)s %(name)s: %(message)s")

from ddtrace.propagation.http import HTTPPropagator

# 1. Datadog headers: value is dropped and reported as a decoding error
ctx = HTTPPropagator.extract({
    "x-datadog-trace-id": "1234",
    "x-datadog-parent-id": "5678",
    "x-datadog-sampling-priority": "2",
    "x-datadog-origin": "synthetics",
    "x-datadog-tags": "_dd.p.dm=-15",
})
print("datadog headers  ->", ctx._meta)

# 2. Same value over W3C tracestate: kept, no warning
ctx = HTTPPropagator.extract({
    "traceparent": "00-000000000000000000000000000004d2-000000000000162e-01",
    "tracestate": "dd=s:2;t.dm:-15",
})
print("tracecontext     ->", ctx._meta)

Run with DD_TRACE_LOGGING_RATE=0 to see the warning on every occurrence.

Error Logs

WARNING ddtrace.internal.sampling: failed to decode _dd.p.dm: '-15'

Libraries in Use

Not integration specific — reproduces with the core tracer only (ddtrace==4.12.0), no instrumentation required.

Operating System

Darwin 25.5.0 (platform independent)

[danez]

We have exactly the same issue, and it also started around the exact same time.