4.10.4

Bug Fixes

• LLM Observability: Fixes agentless export dropping data on the us3, us5, ap1, and ap2 Datadog sites. This affected customers on these sites when no Datadog Agent was running or agentless export was explicitly enabled (DD_LLMOBS_AGENTLESS_ENABLED=1).

• runtime metrics: Fixes an issue where runtime metrics were missing container and orchestrator tags (such as pod_name) on hosts using cgroup v2.

• IAST: A crash that could happen at interpreter teardown has been fixed.

• profiling: A rare crash that could happen after fork in fork-based applications has been fixed.

• tracing: Fixes a bug where running ddtrace-run caused a traceback on keyboard interrupt.

4.11.0rc2

Upgrade Notes

• LLM Observability: when APM and LLMObs are both enabled, the APM trace writer now uses the v0.4 trace API version (v0.5 cannot carry the LLMObs span data). Setting DD_TRACE_API_VERSION=v0.5 with LLMObs enabled logs a warning and downgrades to v0.4. No user action is required.

New Features

• LLM Observability: LLM span data can now be exported to the Datadog Agent over the APM trace, and is reliably delivered even when the APM trace is not sampled. This has no effect on APM sampling decisions or billing.

• This change adds the ability to override a manually installed tracer for Single Step Instrumentation. When this is set, all ddtrace packages will be preferred over the user installed packages. For example, wrapt, bytecode, and others. To ensure the injected library takes precedence, DD_INJECT_EXPERIMENTAL_OVERRIDE_USER_DDTRACE=true can be added to the environment.

Bug Fixes

• LLM Observability: Fixes agentless export dropping data on the us3, us5, ap1, and ap2 Datadog sites. This affected customers on these sites when no Datadog Agent was running or agentless export was explicitly enabled (DD_LLMOBS_AGENTLESS_ENABLED=1).

• tracing: Fixes a race condition where extra service names could be silently dropped from Remote Configuration /v0.7/config payloads in multi-threaded applications (e.g. uWSGI).

• code origin: fixed an issue that could have caused pytest to crash internally when inspecting the call stack from an exception thrown by a view function when Code Origin is enabled.

• LLM Observability: Fixes an issue where spans with very large JSON depth nested fields were being submitted but dropped by Datadog. The LLM Observability integration now detects nested fields that exceed the allowed depth and stringifies them, ensuring spans will not be dropped due to JSON depth limits in Datadog.

• IAST: A crash that could happen at interpreter teardown has been fixed.

• protobuf: This fix resolves an issue where Data Streams Monitoring schema tags were missing from protobuf spans due to incorrect patching of generated message classes.

4.8.9

Bug Fixes

• runtime metrics: Fixes an issue where runtime metrics were missing container and orchestrator tags (such as pod_name) on hosts using cgroup v2.

• code origin: fixed an issue that could have caused pytest to crash internally when inspecting the call stack from an exception thrown by a view function when Code Origin is enabled.

• IAST: A crash that could happen at interpreter teardown has been fixed.

• profiling: A rare crash that could happen after fork in fork-based applications has been fixed.

4.10.3

Bug Fixes

• CI Visibility: fix the default HTTP timeout for backend requests from 15 seconds to 30 seconds, and add the DD_CIVISIBILITY_BACKEND_API_TIMEOUT_MILLIS environment variable (previously missing) to override it. The value is expressed in milliseconds (e.g. 60000 for 60 seconds), consistent with the Java tracer. The same timeout now applies uniformly to all backend requests, including skippable test fetches.

• pydantic_ai: Fixes APM span naming so the operation name is the generic category (pydantic_ai.tool / pydantic_ai.agent) and the resource name is the specific tool or agent name, matching Datadog APM convention. This restores per-tool and per-agent grouping on APM service and resource pages. LLM Observability views are unaffected.

• sqlalchemy: Fixes duplicate SQLAlchemy event listeners when trace_engine() is called repeatedly for the same engine.

• LLM Observability: Fixes an issue where spans with very large JSON depth nested fields were being submitted but dropped by Datadog. The LLM Observability integration now detects nested fields that exceed the allowed depth and stringifies them, ensuring spans will not be dropped due to JSON depth limits in Datadog.

4.9.1

Bug Fixes

• AAP: This fix resolves an issue where the AppSec body-parsing hook consumed the websocket.connect ASGI message, causing ASGI/FastAPI WebSocket connections to fail with HTTP 500 w
  hen AppSec was enabled.

• CI Visibility: fix the default HTTP timeout for backend requests from 15 seconds to 30 seconds, and add the DD_CIVISIBILITY_BACKEND_API_TIMEOUT_MILLIS environment variable (previ
  ously missing) to override it. The value is expressed in milliseconds (e.g. 60000 for 60 seconds), consistent with the Java tracer. The same timeout now applies uniformly to all ba
  ckend requests, including skippable test fetches.

• tracing: Fixes a race condition where extra service names could be silently dropped from Remote Configuration /v0.7/config payloads in multi-threaded applications (e.g. uWSGI).

• LLM Observability: Fixes an issue where reasoning_content was missing from streamed chat completions in the OpenAI and LiteLLM integrations when an OpenAI-compatible reasoning pr
  ovider (e.g. DeepSeek, Qwen) emitted delta.reasoning_content chunks. The aggregated message now captures reasoning text in the output message, matching non-streaming behavior.

• Fixed an issue that could have caused some timers, like the one responsible for Symbol Database uploads, to fire repeatedly after the first execution.

• internal: This fix resolves a memory leak where reference cycles through PeriodicThread callbacks were invisible to Python's cyclic garbage collector and could accumulate when th
  reads used bound methods as targets.

• profiling: Fixes a memory leak in native frame tracking caused by unbounded native call-site metadata growth.

• SCA: This fix resolves an issue where unresolved runtime reachability targets could accumulate across Software Composition Analysis updates, causing resident memory usage to grow o
  ver time.

• sqlalchemy: Fixes duplicate SQLAlchemy event listeners when trace_engine() is called repeatedly for the same engine.

• dynamic instrumentation: fixes an issue where the Symbol Database uploader sends empty payloads on a recurring timer.

• code origin: fixed an issue that could have caused pytest to crash internally when inspecting the call stack from an exception thrown by a view function when Code Origin is enabled

4.8.8

Bug Fixes

• Fixed an issue that could have caused some timers, like the one responsible for Symbol Database uploads, to fire repeatedly after the first execution.

• internal: This fix resolves a memory leak where reference cycles through PeriodicThread callbacks were invisible to Python's cyclic garbage collector and could accumulate when threads used bound methods as targets.

• profiling: Fixes a memory leak in native frame tracking caused by unbounded native call-site metadata growth.

• SCA: This fix resolves an issue where unresolved runtime reachability targets could accumulate across Software Composition Analysis updates, causing resident memory usage to grow over time.

• sqlalchemy: Fixes duplicate SQLAlchemy event listeners when trace_engine() is called repeatedly for the same engine.

• dynamic instrumentation: fixes an issue where the Symbol Database uploader sends empty payloads on a recurring timer.

4.10.2

Bug Fixes

• LLM Observability: This fix resolves an issue in the Claude Agent SDK integration where a span's error message showed an uncategorized unknown error category from the upstream Claude Agent SDK instead of a descriptive API error. The integration now surfaces the detailed error message from the assistant message content.

• tracing: Fixes a race condition where extra service names could be silently dropped from Remote Configuration /v0.7/config payloads in multi-threaded applications (e.g. uWSGI).

• code origin: fixed an issue that could have caused pytest to crash internally when inspecting the call stack from an exception thrown by a view function when Code Origin is enabled.

• LLM Observability: Resolves an issue where non-string tag values passed to LLMObs.annotate(tags=...) could cause spans to be dropped during ingestion.

• LLM Observability: Fixes provider mis-attribution on openai spans when an OpenAI (or AsyncOpenAI) client and an AzureOpenAI (or AsyncAzureOpenAI) client are instantiated at the same time. Provider is now determined per-call rather than from the most recently constructed client.

4.11.0rc1

Upgrade Notes

• flask: Requests served under a non-empty WSGI SCRIPT_NAME now expose the client-hit resource on a new flask.resource.full tag (e.g. GET /api/v2/users). The span resource and flask.url_rule tag are unchanged. The tag is only set when its value would differ from span.resource.

• flask: API endpoint discovery now reports every HTTP method the framework serves — including Werkzeug's auto-added HEAD for any GET route and Flask's auto-handled OPTIONS for every route — not just the methods listed in methods=[...].

New Features

• tracing: collect the x-datadog-endpoint-scan and x-datadog-security-test HTTP request headers on service entry spans unconditionally as http.request.headers.x-datadog-endpoint-scan and http.request.headers.x-datadog-security-test tags. These markers identify Datadog-originated endpoint scans and security tests so the API inventory pipeline can distinguish scan/test traffic from real user traffic. The headers are tagged regardless of DD_TRACE_HEADER_TAGS configuration or AppSec enablement, and are not propagated to downstream services.

• Database Monitoring (DBM) propagation supports dynamic_service as a new DD_DBM_PROPAGATION_MODE value. Set DD_DBM_PROPAGATION_MODE=dynamic_service to inject DBM service metadata and the SQL base hash without injecting trace context.

• ai_guard: add AI Guard evaluation support to the Anthropic SDK Messages instrumentation. Both non-streaming and streaming requests and non-streaming responses are evaluated through the configured AI Guard client (covering Messages.create / Messages.stream and their async and Beta variants), and evaluation is automatically skipped when a framework integration (LangChain) is already evaluating the same call.

• aiokafka: Adds kafka.partition and kafka.message_offset tags to the producer span once the broker acknowledges the send. The partition reflects the partition the broker actually assigned to the message, which may differ from the partition the caller requested. Mirrors the behavior added to the Java tracer in DataDog/dd-trace-java#11107.

• ASM: This introduces the _dd.appsec.normalized_route span tag for FastAPI and Starlette request spans when API Security is enabled. The tag follows RFC-1103 and provides a per-request, framework-agnostic representation of the matched route — converter types are stripped, multi-parameter URL segments are combined with +, path catch-all parameters are emitted as a single tail element, and trailing slashes are preserved as declared. Mount-prefixed sub-application routes are reported with their full assembled path.

• ASM: Extends _dd.appsec.normalized_route span tag support to Flask request spans when API Security is enabled. Flask / Werkzeug <converter:name> route syntax is normalized following RFC-1103: converter types are stripped, multi-parameter URL segments (e.g. /<first>.<last>/) are combined with +, <path:name> catch-all parameters are emitted as a single tail element, and trailing slashes are preserved as declared. Routes served through DispatcherMiddleware sub-apps are reported with their full assembled path (mount prefix included).

• LLM Observability: The claude_agent_sdk integration now emits span links between step, LLM, and tool spans so multi-step traces render the sequencing between LLM calls and tool calls.

• aws_durable_execution_sdk_python: Adds distributed tracing for durable workflows across suspend/resume cycles, so a workflow that pauses and resumes in a later invocation appears as a single connected trace. See aws_durable_execution_sdk_python for details and opt-out configuration.

• AAP: API endpoint discovery now covers Flask sub-applications mounted via werkzeug.middleware.dispatcher.DispatcherMiddleware. Sub-app routes are reported with their full mounted path.

• google_cloud_pubsub: This introduces Data Streams Monitoring (DSM) context propagation for the Google Cloud Pub/Sub integration. Producer publish operations inject the DSM pathway context into message attributes, and subscriber callbacks extract it and record a consume checkpoint. To enable, set DD_DATA_STREAMS_ENABLED=true.

• LLM Observability: automatically tags spans and experiments with git.commit.sha and git.repository_url. Values come from DD_GIT_COMMIT_SHA / DD_GIT_REPOSITORY_URL or the main package's Project-URL metadata, falling back to running git against the current working directory. Honors DD_TRACE_GIT_METADATA_ENABLED and user-supplied tags with the same keys.

• LLM Observability: experiment evaluators and summary evaluators can now return a MultiEvaluatorResult to emit multiple named evaluation metrics from a single evaluator call. Each sub-value may itself be an EvaluatorResult carrying its own reasoning, assessment, metadata, and tags. By default emitted metric labels are prefixed with the evaluator's name ("<evaluator_name>-<key>"); pass prefix=False to emit raw keys.

• LLM Observability: Add LLMObs.pull_experiment(experiment_id) to fetch a previously-run experiment from the Datadog backend by UUID. The returned SyncExperiment has its .result populated and is ready for downstream inspection (e.g. .as_dataframe()) without re-executing the original task.

• LLM Observability: task and dataset are now optional when creating an experiment via LLMObs.experiment() / LLMObs.async_experiment(), supporting pull-based workflows. Calling run() without providing task and dataset raises a ValueError.

• LLM Observability: When a tool definition with a version field is set on a parent LLM span via tool_definitions, the resolved tool version is now also written onto manually-started child tool spans (created via LLMObs.tool()) as meta.tool.version. Previously, this propagation only occurred for tool spans produced by auto-instrumented integrations (OpenAI, LangChain, OpenAI Agents, ReAct) through the LinkTracker dispatch mechanism.

• LLM Observability: When a tool definition with a version field is set on a parent LLM span via tool_definitions, the resolved tool version is now also written onto the corresponding child tool span as meta.tool.version. This allows the LLM Observability UI to aggregate tool version counts and filter tool spans by version without correlating to the parent LLM span.

• profiling: The fast stack profiler optimisation is now enabled by default.

• Profiling: The heap profiler can now track allocations made through PYMEM_DOMAIN_MEM (PyMem_Malloc/Calloc/Realloc) in addition to the existing PYMEM_DOMAIN_OBJ coverage when DD_PROFILING_MEMORY_MEM_DOMAIN_ENABLED=true is set. This support is effective on Python 3.12 and newer, making some previously invisible allocations visible in heap profiles in those environments.

• ray: Added DD_ML_JOB_ENV — a single opt-in env var for forwarding configuration into Ray job workers. Set it on the dashboard process as a semicolon-separated list of KEY:VALUE pairs, e.g. DD_SERVICE:my-svc;DD_AGENT_HOST:10.0.0.1, and workers receive them verbatim as DD_SERVICE=my-svc, DD_AGENT_HOST=10.0.0.1, etc.

• CI Visibility: Adds ddtrace.testing.logs.DDTestLogsHandler, a public logging.Handler for shipping log records to the Datadog logs intake correlated with CI Visibility test traces, intended for use in subprocesses that execute tests outside of the main pytest process. Also adds ddtrace.testing.logs.CorrelationFilter, a base class for stamping dd.trace_id / dd.span_id onto log records, and a ready-made ddtrace.testing.logs.ThreadLocalCorrelationFilter for the common thread-local case. Subclass CorrelationFilter to plug in other concurrency models (asyncio ContextVar, global state, etc.).

  Example:

  from pydantic_evals.evaluators import EqualsExpected import logging

  from ddtrace.testing.logs import DDTestLogsHandler from ddtrace.testing.logs import ThreadLocalCorrelationFilter

  handler = DDTestLogsHandler(service="my-service") correlation = ThreadLocalCorrelationFilter() handler.addFilter(correlation) logging.getLogger().addHandler(handler)

  while True:
  job = queue.get() correlation.set_context(trace_id=job.trace_id, span_id=job.span_id) run_test(job.item)

  handler.close()

Bug Fixes

• AAP: This fix resolves an issue where the AppSec body-parsing hook consumed the websocket.connect ASGI message, causing ASGI/FastAPI WebSocket connections to fail with HTTP 500 when AppSec was enabled.

• aiokafka: Collects the Kafka cluster ID when tracing aiokafka producers and consumers and forwards it to Data Streams Monitoring, matching the behavior of the confluent_kafka integration.

• aap: Adds instrumentation telemetry distributions for WAF and RASP execution durations, including waf.duration, waf.duration_ext, rasp.duration, and rasp.duration_ext.

• azure_cosmos: This fix resolves an issue where cosmosdb.query span resource names included per-item identifiers (such as document, user, and permission ids) from the request URI, which caused unbounded resource cardinality. The integration now redacts those ids by replacing them with ? while keeping the database (dbs) and collection (colls) names intact. For example, Read /dbs/myDb/colls/myColl/docs/item1 is now reported as Read /dbs/myDb/colls/myColl/docs/?.

• CI Visibility: fix the default HTTP timeout for backend requests from 15 seconds to 30 seconds, and add the DD_CIVISIBILITY_BACKEND_API_TIMEOUT_MILLIS environment variable (previously missing) to override it. The value is expressed in m...

4.10.1

Bug Fixes

• internal: Fixed an issue that could have caused some timers, like the one responsible for Symbol Database uploads, to fire repeatedly after the first execution.

• internal: This fix resolves a memory leak where reference cycles through PeriodicThread callbacks were invisible to Python's cyclic garbage collector and could accumulate when threads used bound methods as targets.

• profiling: Fixes a memory leak in native frame tracking caused by unbounded native call-site metadata growth.

• SCA: This fix resolves an issue where unresolved runtime reachability targets could accumulate across Software Composition Analysis updates, causing resident memory usage to grow over time.

4.10.0

Bug Fixes

• AAP: This fix resolves an issue where the AppSec body-parsing hook consumed the websocket.connect ASGI message, causing ASGI/FastAPI WebSocket connections to fail with HTTP 500 when AppSec was enabled.

• LLM Observability: Fixes an issue where reasoning_content was missing from streamed chat completions in the OpenAI and LiteLLM integrations when an OpenAI-compatible reasoning provider (e.g. DeepSeek, Qwen) emitted delta.reasoning_content chunks. The aggregated message now captures reasoning text in the output message, matching non-streaming behavior.

• dynamic instrumentation: fixes an issue where the Symbol Database uploader sends empty payloads on a recurring timer.

Other Changes

• LLM Observability: when LLMObs is enabled in agentless mode (Datadog Agent not reachable or with DD_LLMOBS_AGENTLESS_ENABLED=1), APM traces are now exported agentlessly to Datadog's intake. This should not change user-facing behavior: both APM and LLMObs spans remain visible in the UI; LLMObs spans are simply no longer shipped separately for agentless users. Note that setting DD_APM_TRACING_ENABLED=false takes higher precedence and will result in LLMObs span events shipping separately as existing behavior.