DRIVERS-3454 Support trace context propagation to the server

[nhachicha]

DRIVERS-3454

Please complete the following before merging:

• Is the relevant DRIVERS ticket in the PR title?

• Update changelog.
• Test changes in at least one language driver JAVA-6222 Support trace context propagation to the server mongo-java-driver#2022.
• Test these changes against all server versions and topologies (including standalone, replica set, and sharded
  clusters).

[Copilot]

Pull request overview

Adds specification support for propagating OpenTelemetry trace context from drivers to MongoDB servers via a new OP_MSG telemetry section, gated by wire version (MongoDB 9.0 / maxWireVersion >= 29).

Changes:

• Extend the wire version feature list to include the new telemetry section support in MongoDB 9.0 / wire version 29.
• Specify how drivers propagate traceparent to the server using an OP_MSG Payload Type 3 telemetry section.
• Update the OP_MSG specification to define Payload Type 3 semantics and constraints.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
source/wireversion-featurelist/wireversion-featurelist.md	Adds MongoDB 9.0 / wire version 29 entry describing the OP_MSG telemetry section feature.
source/open-telemetry/open-telemetry.md	Specifies trace context propagation rules (when to attach, validation rules, gating).
source/message/OP_MSG.md	Defines Payload Type 3 section structure, constraints, and changelog entry.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

source/message/OP_MSG.md:170

• Grammar: use "a uint8" (not "an uint8") since it's pronounced "you-int-eight". This sentence was touched in this change, so it's a good time to correct it.

Each message contains one or more sections. A section is composed of an uint8 which determines the payload's type, and a

[codeowners-service-app]

Assigned vbabanin for team dbx-spec-maintainers-connection-establishment because blink1073 is out of office.

[jyemin]

Prose changes LGTM.

1. There's an OP_MSG.md merge conflict. I'd like to review again after it's resolved
2. There are no tests. Is that deliberate or do you plan to add a test plan?

[nhachicha]

@jyemin

There are no tests. Is that deliberate or do you plan to add a test plan?

TL;DR: No plan to add tests. The wire-level assertions aren't expressible cross-driver: the section is invisible to command monitoring, and server spans are only observable with special mongod startup parameters driver CI doesn't provision.

• How Java (reference implmenetation) was tested locally

Server spans are only observable by controlling the mongod process. It must be started with --setParameter opentelemetryTraceDirectory=<dir> plus --setParameter featureFlagOtelTraceSampling=true and --setParameter 'openTelemetryTracingSampling={defaultSampling: {samplingFactor: 1.0}}'. Spans are then exported as OTLP JSON files written to that directory on the server's filesystem, flushed on the batch-export interval (default 1s).

Verification means running the traced operation, waiting past the flush interval, then reading those files and matching the exported span's traceId/parentSpanId against the client span.

[comandeo-mongo]

What is the tracestate suffix? I couldn't find it here

[nhachicha]

Right tracestate is a separate W3C header, not part of traceparent; Removed the mention in 1304087

[comandeo-mongo]

Is the parent_id here an id of the current command span? This might be worth clarifying.

[nhachicha]

Yes, parent-id is the W3C field name, seen from the receiver's side: it carries the span id of the sender's current span, which becomes the parent of the spans the receiver creates

[jyemin]

OK. At first I thought there was no coverage at all, but the existing OTel unified tests do offer some: Claude informs me that the server validates the section during OP_MSG parse regardless of build or feature flags, so a malformed section fails those tests on 9.0+, and a missing wire version gate in the driver will presumably cause a hard failure on servers that don't support the new section at all.

Among the things that leaves uncovered (there may be more):

1. A driver that never attaches the section at all.
2. A section attached to commands that should not include one (auth, monitoring).
3. A valid-but-wrong traceparent, e.g. the operation span's context instead of the command span's.
4. Retries reusing one parent-id across attempts instead of one per attempt.

Server spans are only observable with special mongod startup parameters driver CI doesn't provision.

That's true now, but it doesn't have to remain so. We could create a mongo-orchestration/mongo-runner configuration that enables Otel tracing and invoke it in CI.

Once we do that, there seems to be two simpler ways to verify than the one you noted:

1. serverStatus metric otelTracingSampler.externalSpan.rateLimiter.successfulAdmissions: Asserting on the count could cover gaps one and two.

2. HTTP endpoint: Start mongod with opentelemetryHttpEndpoint pointed at an HTTP listener started by a test runner, capturing all the events. Asserting on the delivered events could cover all four gaps.

I think it's worth adding a prose test that makes use of one or both of these verification options, preferably the second as it provides more coverage.

[nhachicha]

@jyemin

so a malformed section fails those tests on 9.0+, and a missing wire version gate in the driver will presumably cause a hard failure on servers that don't support the new section at all.

The rejection is server-side and safe for the server: pre-9.0 servers error on the unknown section kind and close that connection (the existing OP_MSG behavior for unknown payload types), and 9.0+ servers uassert on a malformed payload. These are ordinary request-scoped errors, the server does not crash, and other connections are unaffected.

A driver that never attaches the section at all.

The section is optional by design, so there's no functional harm in either direction: against pre-9.0 servers nothing is sent, and on 9.0+ OP_MSG simply permits zero telemetry sections.

A section attached to commands that should not include one (auth, monitoring).

That would be a driver violating the existing OTel spec, which already prohibits span creation for security-sensitive commands (and monitoring connections carry no spans). If a driver misbehaves anyway, the server still parses the traceparent and may link a server span to it, subject to its sampling/rate limiting — nothing breaks, but the spans shouldn't exist.

A valid-but-wrong traceparent, e.g. the operation span's context instead of the command span's.

The server can't detect this: any structurally valid (W3C-conformant) traceparent is accepted and used as the parent. The damage shows up in the collector, a server span parented to an id that was never exported (orphaned), or attached to the operation span instead of the command span. That's a driver spec violation invisible to everything except end-to-end span inspection.

Retries reusing one parent-id across attempts instead of one per attempt.

This is precisely why the spec attaches server spans to command spans rather than the operation: the normal case is one operation issuing multiple commands (retry attempts, getMores etc.), each of which should fold under the operation as its own subtree. A driver reusing one parent-id across attempts would hang every server span off a single node flattening the tree and misreport causality, again, only observable from the exported spans.

That's true now, but it doesn't have to remain so. We could create a mongo-orchestration/mongo-runner configuration that enables OTel tracing and invoke it in CI.

Agreed, already thought about it: I have a WIP in drivers-evergreen-tools adding an opt-in --otel flag to drivers_orchestration.py. It launches every mongod/mongos with the file-exporter (while HTTP requires every driver to embed a listener plus an OTLP-protobuf decoder)

Sample usage once it lands:

# provision an OTel-enabled cluster (drivers-evergreen-tools)
OTEL=1 MONGODB_VERSION=latest TOPOLOGY=replica_set bash .evergreen/run-orchestration.sh
source mo-expansion.sh    # exports OTEL_TRACE_DIR

# driver test suite gates the prose test on the variable (skipped when unset)
./gradlew :driver-sync:test --tests '<ProseTest>' \
    -Dorg.mongodb.test.uri="$MONGODB_URI" \
    -Dorg.mongodb.test.otel.trace.dir="$OTEL_TRACE_DIR"

I'd track this as separate tickets in both repos (drivers-evergreen-tools feature and one for the spec's prose test that consumes it) so this PR's isn't blocked on tooling.