feat(profiling): native-heap live-heap cleanup (ABI, build env, auditwheel) (PROF-15423)

[vlad-scherbich]

Description

Multiple cleanup to close off the native-heap feature code commits in Python profiler.

Changes

• Default Cargo feature refactor (wrapper live-heap as the default-on gate for upstream).
• Capability ABI (ddtrace_heap_gotter_live_heap_enabled) + Python query + mode log (live-heap vs allocation-only).
• Fold DD_PROFILING_NATIVE_HEAP_BUILD into DD_PROFILING_NATIVE_HEAP_ENABLED.
• auditwheel .so.debug stash/restore.
• Release note for the stack.

Testing

• Updated unit tests
• CI

[cit-pr-commenter-54b7da]

Circular import analysis

⚠️ Existing circular imports

There are 5 circular imports that already exist on the base branch and have not been changed by this PR.

ddtrace.contrib.internal.django.patch -> ddtrace.contrib.internal.django.response -> ddtrace.contrib.internal.django.patch
ddtrace.contrib.internal.pytorch._distributed -> ddtrace.contrib.internal.pytorch._rank_root -> ddtrace.contrib.internal.pytorch._distributed
ddtrace.errortracking._handled_exceptions.bytecode_injector -> ddtrace.errortracking._handled_exceptions.callbacks -> ddtrace.errortracking._handled_exceptions.collector -> ddtrace.errortracking._handled_exceptions.bytecode_reporting -> ddtrace.errortracking._handled_exceptions.bytecode_injector
ddtrace.llmobs -> ddtrace.llmobs._evaluators -> ddtrace.llmobs._evaluators.format -> ddtrace.llmobs._experiment -> ddtrace.llmobs
ddtrace.appsec._asm_request_context -> ddtrace.appsec._iast._iast_request_context_base -> ddtrace.appsec._iast._iast_env -> ddtrace.appsec._iast.reporter -> ddtrace.appsec._exploit_prevention.stack_traces -> ddtrace.appsec._asm_request_context

[cit-pr-commenter-54b7da]

Codeowners resolved as

Resolved from the full PR diff against main using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

.gitlab/scripts/build-wheel-helpers.sh                                  @DataDog/python-guild @DataDog/apm-core-python
ddtrace/internal/datadog/profiling/heap_gotter/__init__.py              @DataDog/profiling-python
ddtrace/internal/settings/profiling.py                                  @DataDog/profiling-python
ddtrace/profiling/profiler.py                                           @DataDog/profiling-python
releasenotes/notes/native-heap-profiling-live-heap-336bc489c93989da.yaml  @DataDog/apm-python
setup.py                                                                @DataDog/python-guild
src/native_heap_gotter/Cargo.toml                                       @DataDog/profiling-python
src/native_heap_gotter/lib.rs                                           @DataDog/profiling-python
tests/profiling/test_native_heap_gotter.py                              @DataDog/profiling-python
tests/profiling/test_profiling_config.py                                @DataDog/profiling-python

[datadog-datadog-prod-us1-2]

 

✨ Unblock PR with BitsAI

⚠️ Warnings

🚦 9 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-py | build linux serverless: [amd64, cp315-cp315, v113741238-d2b8243-manylinux2014_x86_64, 1]     

DataDog/apm-reliability/dd-trace-py | build linux serverless: [amd64, cp315-cp315, v126532274-233089d-musllinux_1_2_x86_64, 1]     

DataDog/apm-reliability/dd-trace-py | build linux serverless: [arm64, cp315-cp315, v113741357-d2b8243-manylinux2014_aarch64, 1]     

View all 9 failed jobs.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 7d2117f | Docs | Datadog PR Page | Give us feedback!}

[pr-commenter]

Benchmarks

Benchmark execution time: 2026-08-16 13:20:26

Comparing candidate commit 7d2117f in PR branch vlad/native-heap-gotter-freeprobe-pr-e with baseline commit e5c63be in branch main.

📊 Benchmarking dashboard

Found 0 performance improvements and 8 performance regressions! Performance is the same for 614 metrics, 10 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

• 🟩 = significantly better candidate vs. baseline
• 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:httppropagationinject-ids_only

• 🟥 execution_time [+2.362µs; +2.541µs] or [+10.919%; +11.749%]

scenario:iastaspects-lower_aspect

• 🟥 execution_time [+56.094µs; +60.300µs] or [+24.348%; +26.174%]

scenario:iastaspects-modulo_noaspect

• 🟥 execution_time [+24.244µs; +31.475µs] or [+7.440%; +9.659%]

scenario:iastaspectsospath-ospathbasename_aspect

• 🟥 execution_time [+143.178µs; +150.806µs] or [+37.332%; +39.321%]

scenario:iastaspectssplit-rsplit_aspect

• 🟥 execution_time [+15.726µs; +21.824µs] or [+11.253%; +15.616%]

scenario:span-start

• 🟥 execution_time [+1.121ms; +1.293ms] or [+7.539%; +8.697%]

scenario:telemetryaddmetric-1-count-metric-1-times

• 🟥 execution_time [+472.411ns; +515.209ns] or [+17.977%; +19.606%]

scenario:tracer-small

• 🟥 execution_time [+30.302µs; +32.820µs] or [+8.905%; +9.645%]

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:coreapiscenario-context_with_data_listeners

• unstable execution_time [-839.814ns; +642.523ns] or [-7.609%; +5.822%]

scenario:coreapiscenario-core_dispatch_1_listener

• unstable execution_time [-35.748ns; +30.101ns] or [-5.849%; +4.925%]

scenario:coreapiscenario-core_dispatch_50_listeners

• unstable execution_time [-1843.413ns; +1456.273ns] or [-10.760%; +8.500%]

scenario:coreapiscenario-core_dispatch_exception_listeners

• unstable execution_time [-1179.742ns; +1267.197ns] or [-9.170%; +9.850%]

scenario:coreapiscenario-core_dispatch_listeners

• unstable execution_time [-351.638ns; +300.447ns] or [-9.541%; +8.152%]

scenario:coreapiscenario-core_dispatch_no_args_listeners

• unstable execution_time [-265.384ns; +246.825ns] or [-9.045%; +8.412%]

scenario:coreapiscenario-core_dispatch_with_results_1_listener

• unstable execution_time [-123.844ns; +25.683ns] or [-10.380%; +2.153%]

scenario:coreapiscenario-core_dispatch_with_results_50_listeners

• unstable execution_time [-3698.236ns; +4348.603ns] or [-9.104%; +10.705%]

scenario:coreapiscenario-core_dispatch_with_results_listeners

• unstable execution_time [-807.869ns; +731.792ns] or [-9.902%; +8.970%]

scenario:packagesupdateimporteddependencies-import_many_stdlib_cached

• unstable execution_time [-62.683µs; +57.468µs] or [-9.830%; +9.012%]

[scottgerring]

I think it would be good to also have a runtime knob to disable live heap - not do the GOT table override for frees. This still saves the user cost, because nothing will ever bind on the free side, and I assume our regular build of dd-trace-py will have the flag set (??).

By omitting the USDT altogether in the binary, the profiler gets the hint it doesn't need to track allocations at all for a process, so its kind of a nice optimisation for folks who really really sweat resources, but I figure the regular user is going to probably going to want to tune at runtime

[vlad-scherbich]

@codex review

[Copilot]

Pull request overview

Completes the producer-side “live heap” native heap profiling stack by exposing whether the loaded heap-gotter cdylib was compiled with live-heap tracking (alloc + free), surfacing that in the Python activator and profiler logs, and updating build/release-note plumbing accordingly.

Changes:

• Add a stable C ABI symbol (ddtrace_heap_gotter_live_heap_enabled) and a Python wrapper (heap_gotter.live_heap_enabled()) to report live-heap capability from the loaded cdylib.
• Improve profiler arming observability by logging whether “live-heap” vs “allocation-only” was armed.
• Update wheel build helper script to more robustly stash/remove/reinsert the gotter artifact around auditwheel, and add a consolidated release note.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/profiling/test_native_heap_gotter.py	Extends smoke test to assert live_heap_enabled() is safe when cdylib is absent and returns a bool when present.
src/native_heap_gotter/lib.rs	Exposes ddtrace_heap_gotter_live_heap_enabled() over the C ABI using cfg!(feature = "live-heap").
src/native_heap_gotter/Cargo.toml	Adds crate-local live-heap feature (default-on) intended to mirror upstream live-heap compilation.
releasenotes/notes/native-heap-profiling-live-heap-336bc489c93989da.yaml	Adds a single release note describing the end-to-end experimental native heap profiling feature.
ddtrace/profiling/profiler.py	Logs which native-heap mode was armed based on heap_gotter.live_heap_enabled().
ddtrace/internal/datadog/profiling/heap_gotter/init.py	Defensively binds the new cdylib symbol and exposes live_heap_enabled() fail-closed.
.gitlab/scripts/build-wheel-helpers.sh	Switches to a pure-Python stash/remove flow for the heap-gotter artifact before auditwheel, and reinserts only the runtime .so.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fff0a7ee02

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (1)

ddtrace/internal/datadog/profiling/heap_gotter/init.py:5

• This module docstring uses reStructuredText-style inline literals (...) even though this is an internal module (not Sphinx-rendered). This makes the docstring noisier to read in editors; prefer plain prose for internal docstrings.

"""Activator for native (C/C++) heap allocation profiling via GOT rewriting.

Dlopens ``libdd_heap_gotter`` (see ``src/native_heap_gotter``) and drives:

    bool ddtrace_heap_gotter_install(void);

[KowalskiThomas]

When trying to build native heap profiling from this PR's code, I get several failing jobs (see #19701).

• build linux serverless
• build linux: [amd64, cp311-cp311, v126532274-233089d-musllinux_1_2_x86_64]

error: cannot produce cdylib for `ddtrace-heap-gotter v0.1.0 (/go/src/github.com/DataDog/apm-reliability/dd-trace-py/src/native_heap_gotter)` as the target `x86_64-unknown-linux-musl` does not support these crate types

That doesn't seem intended, since trying to build with heap profiling should probably soft fail if unsupported on a platform, can you look into it?

[vlad-scherbich]

fixed here, and being handled gracefully on #19715

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Suppressed comments (4)

ddtrace/internal/settings/profiling.py:562

• This help text conflates the runtime enable flag with the wheel build requirement. DD_PROFILING_NATIVE_HEAP_ENABLED is used to enable the feature at runtime, but the wheel must also include the gotter cdylib (which is currently opt-in at build time per setup.py). Suggest updating the help string to explicitly say it requires a wheel that ships the native heap gotter artifact (and, if applicable, how that artifact is produced), instead of implying setting the runtime env var affects the wheel contents.

            "DD_PROFILING_NATIVE_HEAP_ENABLED=1. Disabled by default."

releasenotes/notes/native-heap-profiling-live-heap-336bc489c93989da.yaml:5

• The release note implies that setting DD_PROFILING_NATIVE_HEAP_ENABLED=true is sufficient, but the implementation still requires a wheel that actually contains the libdd_heap_gotter artifact (currently opt-in at build time). Please amend the note to mention the wheel/artifact requirement (or adjust build defaults if the intent is to ship this in mainline wheels). Also remove the trailing whitespace after allocations on line 4.

    profiling: Enable collection of samples for native (C/C++) allocations 
    in Python processes. Enable with ``DD_PROFILING_NATIVE_HEAP_ENABLED=true``.
    This setting works for Linux only.

tests/profiling/test_native_heap_gotter.py:33

• This test hard-requires live_heap_enabled() to be True whenever the cdylib is available, but the PR explicitly supports alloc-only builds via --no-default-features (and live_heap_enabled() is documented to return False in that case). To avoid configuration-dependent failures, either (a) assert only that live_heap_enabled() returns a boolean / does not raise, or (b) gate the expected value on an explicit build/test signal (e.g., an env var indicating the wheel was built with live-heap).

        assert heap_gotter.is_installed() is False
        assert heap_gotter.install() is True
        assert heap_gotter.is_installed() is True
        assert heap_gotter.install() is True  # idempotent
        # Default gotter builds enable the live-heap Cargo feature (ddheap:free).
        assert heap_gotter.live_heap_enabled() is True

ddtrace/internal/datadog/profiling/heap_gotter/init.py:17

• The docstring labels DD_PROFILING_NATIVE_HEAP_ENABLED as a 'setup.py build gate' and a runtime gate. The build decision happens at package build time, not at runtime, and users cannot make an already-built wheel include the cdylib by setting a runtime env var. Please reword this section to distinguish: (1) runtime arming (DD_PROFILING_NATIVE_HEAP_ENABLED) vs (2) whether the installed wheel actually contains the gotter artifact (controlled during wheel build).

Two independent runtime gates, both required for GOT to be patched:

* ``DD_PROFILING_NATIVE_HEAP_ENABLED`` — ddtrace install gate (also the
  setup.py build gate). When false, this module is not imported from the
  profiler and ``install()`` is never called.