feat(utils): native OpenTelemetry tracing via dspy.enable_otel_tracing()

[isaacbmiller]

Motivation

DSPy has no first-party, vendor-neutral tracing story. The two existing paths are:

• MLflow (mlflow.dspy.autolog()): built cleanly on DSPy's BaseCallback system, but exports only to MLflow.
• OpenInference (openinference-instrumentation-dspy): vendor-neutral OTLP, but bypasses the callback system entirely and monkey-patches DSPy internals with wrapt (LM.__call__, Predict.forward, Module.__call__, Adapter, Tool, Retrieve, plus custom copyable wrappers so deepcopy doesn't break). It reaches into signature.output_fields, Prediction._store, and __qualname__ dispatch, and has broken across DSPy releases (e.g. [Bug] 2.6.0 Instrumentation issue #7627).

This PR adds native OpenTelemetry emission built on the public callback surface, so any OTLP backend (Jaeger, Grafana, Langfuse, Phoenix via its convention-translation processor, Datadog, ...) works without third-party patching of DSPy internals:

import dspy

dspy.enable_otel_tracing()  # or dspy.configure(callbacks=[dspy.OtelCallback()])

What this looks like

A dspy.ReAct agent with one tool, exported to a local Jaeger via OTLP:

LM spans carry GenAI-semconv attributes, including full structured messages (content capture is on by default, with an opt-out):

Design

• Callback bridge, not per-call-site instrumentation. OtelCallback maps BaseCallback events to spans; parenting rides the OTel context, so DSPy spans nest correctly under surrounding application spans and vice versa.
• OTel GenAI semantic conventions (gen_ai.*) for LM spans (chat {model}, CLIENT kind, provider/model/temperature/max_tokens, gen_ai.input.messages / gen_ai.output.messages, gen_ai.usage.*) and tool spans (execute_tool {name}, gen_ai.tool.*). Since the GenAI conventions are still in Development status, semconv_version=1 is an explicit knob so attribute layouts can evolve without breaking dashboards.
• dspy.* attributes for spans with no GenAI equivalent: modules (dspy.module.type, dspy.module.signature, content-gated dspy.module.input/output), adapters (ChatAdapter.format / .parse spans), and Evaluate (dspy.evaluate.num_examples, dspy.evaluate.score).
• Content capture defaults on (enabling tracing is already an explicit opt-in); disable per callback (capture_content=False) or via the standard OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT env var.
• Optional dependency. Only opentelemetry-api (new dspy[otel] extra), imported lazily inside OtelCallback.__init__; import dspy is unaffected when it isn't installed. Without a configured SDK the API is a no-op.
• Token usage is emitted when LM outputs are a typed LMResponse (explicit LMRequest calls or dspy.context(experimental=True)). The legacy list-output path has no race-free per-call usage source; that needs a callback-payload enrichment and is deliberately left to a follow-up.

ParallelExecutor contextvars fix

ParallelExecutor previously copied only thread_local_overrides into worker threads, so contextvars — the OTel context and the callback system's ACTIVE_CALL_ID — did not propagate, and spans created inside Evaluate/Parallel workers became orphan traces. Workers now run inside contextvars.copy_context() captured at submit time. This also fixes span parenting for other callback-based integrations (e.g. MLflow) across Evaluate/Parallel.

Tests

• tests/utils/test_otel.py (10 tests): span nesting/parenting for ChainOfThought, content gating (param + env var), typed-path gen_ai.usage.*, tool success/error spans with ERROR status + exception events, async calls, Evaluate spans parenting across worker threads, idempotent enable_otel_tracing(), unsupported semconv_version rejection.
• Neighboring suites pass: tests/callback, tests/utils/test_parallelizer.py, tests/evaluate, tests/streaming (72 passed, 30 skipped).
• Verified end-to-end with a real openai/gpt-5-nano ReAct program exported to Jaeger via OTLP (screenshots above).

Not in this PR (planned follow-ups)

• Streaming span coverage (callbacks do not fire around streamed responses).
• Compile/optimizer tracing and gating (pairs with the upcoming optimizer lifecycle hooks).
• Legacy-path token usage via callback payload enrichment.
• Docs page under the observability tutorial.

🤖 Generated with Claude Code