chore(tracing): avoid reentrant SpanData borrow panic

[juanjux]

Description

Dogweb's dd-trace-py 4.14.0rc2 dogfooding exposed a re-entrant PyO3 borrow panic under CI Visibility, Flask, and SQLAlchemy workloads:

pyo3_runtime.PanicException: Already mutably borrowed: PyBorrowError

The failure starts when a non-primitive span tag is converted to a string. In the observed case, the object's __str__ accessed an expired SQLAlchemy attribute, which issued a query and started a nested database span. Starting that span calls context_provider.active().

SpanData._set_attribute previously held a mutable PyO3 borrow for the entire method, including while calling user-defined __str__ and __index__. The native context provider introduced in 4.14 reads the active SpanData with a shared borrow. Re-entering it during string conversion therefore attempted a shared borrow while the mutable borrow was still live, causing PyO3 to panic. When this occurred during a SQLAlchemy transaction, the abrupt exception could also leave the session in the prepared state and cause later tests to fail.

This change performs all potentially re-entrant Python coercion before borrowing SpanData mutably. The mutable borrow is now limited to updating the native attribute map, and replaced Python values are dropped only after that borrow is released. Bulk and default attribute setters follow the same pattern.

Additional Notes

The SQLAlchemy prepared session errors are downstream fallout from the panic, not a separate SQLAlchemy integration issue. This fixes the invalid borrow at its source rather than catching or suppressing PanicException in an integration.

[datadog-prod-us1-3]

 

✨ Unblock PR with BitsAI

⚠️ Warnings

🚦 10 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-py | build linux serverless: [amd64, cp315-cp315, v126532274-233089d-musllinux_1_2_x86_64, 1]     

DataDog/apm-reliability/dd-trace-py | build linux serverless: [arm64, cp315-cp315, v113741357-d2b8243-manylinux2014_aarch64, 1]     

DataDog/apm-reliability/dd-trace-py | build linux: [amd64, cp315-cp315, v113741238-d2b8243-manylinux2014_x86_64]     

View all 10 failed jobs.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

🔄 Datadog auto-retried 1 job - 1 passed on retry

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 8905c27 | Docs | Datadog PR Page | Give us feedback!}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d8e1f22327

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[github-actions]

This change is marked for backport to 4.14 and it does not conflict with that branch.
The command used to test backporting was

git fetch origin 4.14 && git checkout origin/4.14 && git checkout -b backport-19699-to-4.14 && git cherry-pick -x --mainline 1 34109749a48537b81e382a218c3e38aafd1848f6

[pr-commenter]

Benchmarks

Benchmark execution time: 2026-08-14 16:35:06

Comparing candidate commit 8905c27 in PR branch juanjux/fix-reentrant-span-data-borrow with baseline commit 3e902cc in branch main.

📊 Benchmarking dashboard

Found 0 performance improvements and 8 performance regressions! Performance is the same for 610 metrics, 10 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

• 🟩 = significantly better candidate vs. baseline
• 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:httppropagationinject-ids_only

• 🟥 execution_time [+2.441µs; +2.608µs] or [+11.227%; +11.994%]

scenario:iastaspects-repr_aspect

• 🟥 execution_time [+77.472µs; +83.002µs] or [+20.813%; +22.298%]

scenario:iastaspects-upper_aspect

• 🟥 execution_time [+53.078µs; +57.688µs] or [+22.536%; +24.494%]

scenario:iastaspectsospath-ospathbasename_aspect

• 🟥 execution_time [+130.254µs; +138.406µs] or [+32.162%; +34.175%]

scenario:iastaspectssplit-rsplit_aspect

• 🟥 execution_time [+11.599µs; +17.020µs] or [+8.093%; +11.875%]

scenario:span-start

• 🟥 execution_time [+1.359ms; +1.535ms] or [+8.878%; +10.027%]

scenario:telemetryaddmetric-1-count-metric-1-times

• 🟥 execution_time [+522.396ns; +563.948ns] or [+19.762%; +21.334%]

scenario:tracer-small

• 🟥 execution_time [+30.444µs; +32.975µs] or [+9.008%; +9.758%]

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:coreapiscenario-context_with_data_listeners

• unstable execution_time [-760.077ns; +729.318ns] or [-6.894%; +6.615%]

scenario:coreapiscenario-core_dispatch_1_listener

• unstable execution_time [-28.541ns; +37.835ns] or [-4.668%; +6.188%]

scenario:coreapiscenario-core_dispatch_50_listeners

• unstable execution_time [-1839.814ns; +1537.748ns] or [-10.675%; +8.922%]

scenario:coreapiscenario-core_dispatch_exception_listeners

• unstable execution_time [-1278.442ns; +1187.769ns] or [-9.924%; +9.220%]

scenario:coreapiscenario-core_dispatch_listeners

• unstable execution_time [-330.710ns; +323.345ns] or [-8.989%; +8.789%]

scenario:coreapiscenario-core_dispatch_no_args_listeners

• unstable execution_time [-234.703ns; +285.704ns] or [-8.011%; +9.752%]

scenario:coreapiscenario-core_dispatch_with_results_1_listener

• unstable execution_time [-76.319ns; +70.179ns] or [-6.528%; +6.003%]

scenario:coreapiscenario-core_dispatch_with_results_50_listeners

• unstable execution_time [-3778.858ns; +4236.784ns] or [-9.270%; +10.394%]

scenario:coreapiscenario-core_dispatch_with_results_listeners

• unstable execution_time [-875.323ns; +685.747ns] or [-10.643%; +8.338%]

scenario:packagesupdateimporteddependencies-import_many_stdlib_cached

• unstable execution_time [-57.425µs; +63.223µs] or [-9.006%; +9.915%]

[brettlangdon]

Some test coverage missing is _set_attributes re-entrancy, and should we get some tests for _set_default_attributes / set_default_attribute ?

[cit-pr-commenter-54b7da]

Codeowners resolved as

Resolved from the full PR diff against main using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

src/native/span/span_data.rs                                            @DataDog/apm-core-python
tests/tracer/test_span_tags.py                                          @DataDog/apm-sdk-capabilities-python

[cit-pr-commenter-54b7da]

Circular import analysis

⚠️ Existing circular imports

There are 5 circular imports that already exist on the base branch and have not been changed by this PR.

ddtrace.contrib.internal.pytorch._distributed -> ddtrace.contrib.internal.pytorch._rank_root -> ddtrace.contrib.internal.pytorch._distributed
ddtrace.contrib.internal.django.patch -> ddtrace.contrib.internal.django.response -> ddtrace.contrib.internal.django.patch
ddtrace.errortracking._handled_exceptions.bytecode_injector -> ddtrace.errortracking._handled_exceptions.callbacks -> ddtrace.errortracking._handled_exceptions.collector -> ddtrace.errortracking._handled_exceptions.bytecode_reporting -> ddtrace.errortracking._handled_exceptions.bytecode_injector
ddtrace.llmobs -> ddtrace.llmobs._evaluators -> ddtrace.llmobs._evaluators.format -> ddtrace.llmobs._experiment -> ddtrace.llmobs
ddtrace.appsec._asm_request_context -> ddtrace.appsec._iast._iast_request_context_base -> ddtrace.appsec._iast._iast_env -> ddtrace.appsec._iast.reporter -> ddtrace.appsec._exploit_prevention.stack_traces -> ddtrace.appsec._asm_request_context

[cit-pr-commenter-54b7da]

Dependency direction analysis

⚠️ Existing dependency direction violations

There are 255 dependency direction violations that already exist on the base branch and have not been changed by this PR.

Show existing violations (showing 5 of 255 highest severity)

ddtrace.internal.tracemethods -×-> ddtrace.trace  (internal-core -> product:tracing, score=134)
ddtrace.llmobs._integrations.vllm -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=132)
ddtrace.profiling.scheduler -×-> ddtrace.trace  (product:profiling -> product:tracing, score=132)
ddtrace.internal.opentelemetry.trace -×-> ddtrace.trace  (product:opentelemetry -> product:tracing, score=132)
ddtrace.internal.test_visibility.api -×-> ddtrace.trace  (product:ci_visibility -> product:tracing, score=132)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:

uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json

[juanjux]

Some test coverage missing is _set_attributes re-entrancy, and should we get some tests for _set_default_attributes / set_default_attribute ?

@brettlangdon agreed, done.