build(profiling): add native heap-gotter cdylib build (PROF-15750) by vlad-scherbich · Pull Request #19078 · DataDog/dd-trace-py

vlad-scherbich · 2026-07-15T16:37:30Z

Description

Adds a standalone Rust cdylib (src/native_heap_gotter/) wrapping libdatadog's crates.io crate libdd-profiling-heap-gotter 1.0.0 behind a stable, ddtrace-owned C ABI: ddtrace_heap_gotter_install() / _is_installed() (→ bool, idempotent/fork-safe) and _test_hook_hits() (→ u64, test-support feature only, never shipped). Owning the cdylib ([lib] name = "dd_heap_gotter") yields a clean libdd_heap_gotter.so and fixed, unmangled symbols for the future ctypes activator.

Build-gated: built only with DD_PROFILING_NATIVE_HEAP_BUILD=1 on Linux 64-bit. setup.py runs cargo build --release, stages libdd_heap_gotter<EXT_SUFFIX>.so under ddtrace/internal/datadog/profiling/, sets the soname (patchelf), and adds it to package_data only when the flag is set. Off by default → mainline wheels unaffected.
Inert: nothing loads or calls the cdylib yet → zero runtime behavior change. Activation lands in chore(profiling): activate ddheap:alloc and ddheap:free USDTs in native heap gotter (PROF-15753) #19079 (gated by DD_PROFILING_NATIVE_HEAP_ENABLED).

Testing

cargo build --release --manifest-path src/native_heap_gotter/Cargo.toml resolves against crates.io and produces libdd_heap_gotter.so.
Wheel built with DD_PROFILING_NATIVE_HEAP_BUILD=1 (Linux 64-bit) ships the staged .so with the expected soname.
Default build (flag unset / non-Linux / 32-bit) is unchanged and ships no .so.

cit-pr-commenter-54b7da · 2026-07-15T16:38:39Z

Circular import analysis

⚠️ Existing circular imports

There are 6 circular imports that already exist on the base branch and have not been changed by this PR.

Show existing cycles (showing 5 of 6 shortest)

ddtrace.contrib.internal.pytorch._distributed -> ddtrace.contrib.internal.pytorch._rank_root -> ddtrace.contrib.internal.pytorch._distributed
ddtrace.contrib.internal.django.patch -> ddtrace.contrib.internal.django.response -> ddtrace.contrib.internal.django.patch
ddtrace.appsec._common_module_patches -> ddtrace.appsec._listeners -> ddtrace.appsec._common_module_patches
ddtrace.llmobs -> ddtrace.llmobs._evaluators -> ddtrace.llmobs._evaluators.format -> ddtrace.llmobs._experiment -> ddtrace.llmobs
ddtrace.errortracking._handled_exceptions.bytecode_injector -> ddtrace.errortracking._handled_exceptions.callbacks -> ddtrace.errortracking._handled_exceptions.collector -> ddtrace.errortracking._handled_exceptions.bytecode_reporting -> ddtrace.errortracking._handled_exceptions.bytecode_injector

To see all cycles, download the cycles-base.json and cycles-pr.json artifacts from this CI job and run:
uv run --script scripts/import-analysis/cycles.py compare cycles-base.json cycles-pr.json

cit-pr-commenter-54b7da · 2026-07-15T16:38:44Z

Codeowners resolved as

Resolved from the full PR diff against main using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

No remaining files require a CODEOWNERS review.

datadog-datadog-prod-us1-2 · 2026-07-15T16:41:06Z

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🔄 Datadog auto-retried 1 job - 1 passed on retry

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: a7e3bbb | Docs | Datadog PR Page | Give us feedback!}

pr-commenter · 2026-07-15T17:09:32Z

Benchmarks

Benchmark execution time: 2026-08-12 13:54:22

Comparing candidate commit a7e3bbb in PR branch vlad/native-heap-gotter-cdylib-build-pr-a with baseline commit 14ff8ee in branch main.

Found 0 performance improvements and 8 performance regressions! Performance is the same for 613 metrics, 9 unstable metrics.

scenario:httppropagationextract-datadog_tracecontext_tracestate_not_propagated_on_trace_id_no_match

🟥 execution_time [+4.961µs; +5.240µs] or [+7.403%; +7.820%]

scenario:httppropagationinject-ids_only

🟥 execution_time [+2.766µs; +2.940µs] or [+13.175%; +14.002%]

scenario:iastaspects-lstrip_aspect

🟥 execution_time [+57.052µs; +64.965µs] or [+19.586%; +22.303%]

scenario:iastaspects-translate_aspect

🟥 execution_time [+49.391µs; +53.943µs] or [+11.663%; +12.737%]

scenario:iastaspectsospath-ospathbasename_aspect

🟥 execution_time [+102.735µs; +110.742µs] or [+25.496%; +27.483%]

scenario:span-start

🟥 execution_time [+1.412ms; +1.532ms] or [+8.781%; +9.530%]

scenario:telemetryaddmetric-1-count-metric-1-times

🟥 execution_time [+600.949ns; +651.909ns] or [+23.363%; +25.344%]

scenario:telemetryaddmetric-record-1000-metrics

🟥 execution_time [+190.961µs; +226.145µs] or [+9.183%; +10.875%]

vlad-scherbich · 2026-07-28T18:56:09Z

@codex review plz ty

Copilot

Pull request overview

Adds the first (build-only) component of the native heap profiling stack by introducing a standalone Rust cdylib (“heap gotter”) that wraps the published libdd-profiling-heap-gotter crate and stages the resulting shared library into the Python wheel when opt-in build flags are enabled.

Changes:

Introduces a new Rust cdylib crate (src/native_heap_gotter) that exports a small stable C ABI for installation/status/testing hooks.
Extends setup.py to optionally build and package libdd_heap_gotter<EXT_SUFFIX>.so when DD_PROFILING_NATIVE_HEAP_BUILD=1 on Linux x86_64.
Ignores the new crate’s Rust build artifacts in .gitignore.

Reviewed changes

Copilot reviewed 3 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/native_heap_gotter/lib.rs	Exposes stable `extern "C"` entry points wrapping the upstream Rust API.
src/native_heap_gotter/Cargo.toml	Defines the wrapper cdylib crate and pins `libdd-profiling-heap-gotter = 1.0.0` with `default-features = false`.
src/native_heap_gotter/Cargo.lock	Locks Rust dependencies for reproducible builds of the new crate.
setup.py	Adds opt-in build flag, builds/stages the cdylib, and includes it in `package_data` when enabled.
.gitignore	Ignores `src/native_heap_gotter/target*` artifacts.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b8309ae723

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

vlad-scherbich · 2026-07-29T01:17:16Z

@codex re-review

Copilot

Pull request overview

Copilot reviewed 7 out of 9 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

setup.py:1058

cargo build for the heap-gotter crate should use --locked to guarantee the checked-in Cargo.lock is honored and to prevent build-time lockfile updates/drift (especially important for reproducible wheel builds).

        cargo_cmd = [
            "cargo",
            "build",
            "--release",
            "--manifest-path",

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7ddaa1702f

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

emmettbutler

Seems ok. Main concern is binary size, which it sounds like is limited by the opt-in nature of this feature. Hopefully the affected artifact is having its size checked somewhere in CI.

vlad-scherbich · 2026-08-12T13:11:12Z

Seems ok. Main concern is binary size, which it sounds like is limited by the opt-in nature of this feature. Hopefully the affected artifact is having its size checked somewhere in CI.

@emmettbutler

Agreed on the concern. For now, wheels stay unchanged, unless DD_PROFILING_NATIVE_HEAP_BUILD=true. There's no dedicated size check for libdd_heap_gotter* in CI today, but I can add this as a follow up when the feature is GA'ed.

Task: https://datadoghq.atlassian.net/browse/PROF-15749

Add a build-only Rust cdylib (src/native_heap_gotter) that wraps the published crates.io crate libdd-profiling-heap-gotter 1.0.0 and exports a small C ABI (ddtrace_heap_gotter_install / _is_installed / _test_hook_hits). When built (DD_PROFILING_NATIVE_HEAP_BUILD=1, Linux 64-bit) the wheel ships libdd_heap_gotter*.so with a patchelf-set soname; the profiler does not load it yet, so this PR is inert. Allocation-only surface (ddheap:alloc USDT); default-features=false keeps live-heap opt-in and off.

- setup.py: locate the heap-gotter cdylib from cargo JSON artifact output instead of a hard-coded target/release path, so CARGO_TARGET_DIR and --target-dir/--target in DD_CARGO_ARGS are honored. - setup.py: correct the opt-in build rationale now that the gotter is a crates.io version pin (not a moving libdatadog main SHA). - lib.rs: ddtrace_heap_gotter_install() returns the actual install result so a no-op install (no symbols resolved / sampling disabled) is not reported as success while remaining idempotent across fork(). - MANIFEST.in: prune src/native_heap_gotter/target* from sdists. - CI: run cargo fmt/clippy/test on the new crate in "rust ci", trigger profiling_native on its changes, and scan it in the coverage guard.

Annotate the opt-in native heap-gotter cdylib build knobs and helpers in setup.py: the NATIVE_HEAP_GOTTER_CRATE / BUILD_NATIVE_HEAP_GOTTER module globals and the build_heap_gotter / _clean_stale_heap_gotter methods (signatures + locals).

Match the linux wheel CI pattern used for _native: extract .debug sidecars with objcopy/strip at staging time so dogfood wheels ship a stripped libdd_heap_gotter.so, while CI debug-symbols.zip still collects the sidecar.

Switch the native heap gotter cdylib to mirror upstream libdd-profiling-heap-gotter-ffi (libdatadog@56dab857b), expose ddog_* / VoidResult symbols, stage liblibdd_profiling_heap_gotter_ffi artifacts, and add a fail-closed Python ctypes activator with smoke tests.

auditwheel iter_versions crashes on the opt-in heap-gotter cdylib ELF when DD_PROFILING_NATIVE_HEAP_BUILD=1, failing build-linux wheel jobs. Exclude the gotter artifact from repair, matching extract_debug_symbols ignores.

Rename to build(profiling) so changelog/no-changelog is allowed for this inert cdylib packaging change (no customer-facing behavior yet).

Required for feat PR changelog validation after removing no-changelog label.

Build-only native heap-gotter cdylib wiring has no customer-facing release note.

… feature

…el repair

cit-pr-commenter-54b7da · 2026-08-12T13:19:08Z

Dependency direction analysis

⚠️ Existing dependency direction violations

There are 256 dependency direction violations that already exist on the base branch and have not been changed by this PR.

Show existing violations (showing 5 of 256 highest severity)

ddtrace.internal.tracemethods -×-> ddtrace.trace  (internal-core -> product:tracing, score=134)
ddtrace.debugging._signal.tracing -×-> ddtrace.trace  (product:debugging -> product:tracing, score=132)
ddtrace.debugging._signal.model -×-> ddtrace.trace  (product:debugging -> product:tracing, score=132)
ddtrace.llmobs._integrations.base -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=132)
ddtrace.llmobs._integrations.anthropic -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=132)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:
uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json

…(PROF-15753) (#19079) ## Description Makes the heap-gotter cdylib from #19078 reachable: dd-trace-py can now *arm* native (C/C++) heap allocation profiling at profiler startup. All the heavy lifting of attaching uprobes, collecting, symbolizing, uploading is done by the OpenTelemetry eBPF profiler and DD FH profiler out of process. ### Changes * Activator > New `ddtrace.internal.datadog.profiling.heap_gotter`: a small `ctypes` wrapper over the cdylib's two-function C ABI (`install()`, `is_installed()`). The `CDLL` handle is held at module scope for the life of the process and never unloaded, because GOT patching cannot be undone. `install()` is idempotent and fork-safe: children inherit both the mapping and the patched GOT, so a post-fork re-install is a no-op. * Configuration > Adds `DD_PROFILING_NATIVE_HEAP_ENABLED` (boolean, default `false`) as a `native_heap` namespace on `ProfilingConfig`, registered in `supported-configurations.json` and `_supported_configurations.py`. It is a co-requirement, not a standalone switch: arming happens inside profiler startup, and the profiler only starts when `DD_PROFILING_ENABLED=true` (`ddtrace/bootstrap/preload.py` gates on `profiling_config.enabled`), so the native-heap flag alone does nothing. Availability is resolved lazily and only when the flag is on — the disabled path never even imports the activator, let alone `dlopen`s anything. If the flag is on but the cdylib cannot be loaded, the config is forced back off with a warning plus a telemetry error log rather than failing startup. When effectively on, `nativeheap` shows up in the `profiler_config` tag. * Profiler wiring > `_ProfilerInstance._start_service()` calls `heap_gotter.install()` when the feature is enabled, wrapped so that any failure is logged at debug level and swallowed — arming is best-effort and must never prevent the profiler from coming up. Also adds a narrow `mypy.ini` override for `ddtrace.internal.settings.profiling` (the untyped vendored `psutil` calls in the existing heap-sample-size derivation have no stubs in the lint env). ## Testing ### Unit * `tests/profiling/test_native_heap_gotter.py` * `tests/profiling/test_profiling_config.py` — default-off, enable-via-env, and `nativeheap` presence/absence in the `profiler_config` tag string; plus fail-closed assertions on the activator that hold on any platform. ### Staging testing ## Risks No risk for opted-out users. Low risk for opted-in, as there was quite a bit of AB-testing the feature. Co-authored-by: vlad.scherbich <vlad.scherbich@datadoghq.com>

vlad-scherbich added the changelog/no-changelog A changelog entry is not required for this PR. label Jul 15, 2026

vlad-scherbich force-pushed the vlad/native-heap-gotter-cdylib-build-pr-a branch 2 times, most recently from bbd57cb to 852359d Compare July 21, 2026 21:22

vlad-scherbich force-pushed the vlad/native-heap-gotter-cdylib-build-pr-a branch from 683165f to b8309ae Compare July 27, 2026 19:21

vlad-scherbich changed the title ~~feat(profiling): native heap gotter cdylib build (PR A)~~ feat(profiling): native heap gotter cdylib build Jul 27, 2026

This was referenced Jul 27, 2026

chore(profiling): activate ddheap:alloc and ddheap:free USDTs in native heap gotter (PROF-15753) #19079

Merged

feat(profiling): native-heap live-heap cleanup (ABI, build env, auditwheel) (PROF-15423) #19325

Open

vlad-scherbich requested a review from Copilot July 28, 2026 18:55

Copilot started reviewing on behalf of vlad-scherbich July 28, 2026 18:56 View session

Copilot AI reviewed Jul 28, 2026

View reviewed changes

Comment thread setup.py Outdated

Comment thread setup.py Outdated

chatgpt-codex-connector Bot reviewed Jul 28, 2026

View reviewed changes

Comment thread setup.py Outdated

Comment thread .gitignore

Comment thread src/native_heap_gotter/Cargo.toml

Comment thread src/native_heap_gotter/lib.rs Outdated

vlad-scherbich changed the title ~~feat(profiling): native heap gotter cdylib build~~ chore(profiling): native heap gotter cdylib build Jul 29, 2026

vlad-scherbich changed the title ~~chore(profiling): native heap gotter cdylib build~~ feat(profiling): add native heap-gotter cdylib build (opt-in, inert) Jul 29, 2026

vlad-scherbich mentioned this pull request Jul 29, 2026

feat(profiling): prevent double-counting samples allocated with raw malloc (PROF-15768) #19349

Draft

vlad-scherbich requested a review from Copilot July 29, 2026 01:17

Copilot started reviewing on behalf of vlad-scherbich July 29, 2026 01:17 View session

Copilot AI reviewed Jul 29, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Jul 29, 2026

View reviewed changes

Comment thread src/native_heap_gotter/lib.rs

Comment thread src/native_heap_gotter/lib.rs

Comment thread .gitlab/native.yml Outdated

Comment thread setup.py

vlad-scherbich mentioned this pull request Jul 29, 2026

feat(profiling): surface native-heap partition arming via dogstatsd gauge + warning #19376

Closed

5 tasks

vlad-scherbich force-pushed the vlad/native-heap-gotter-cdylib-build-pr-a branch 2 times, most recently from b91736e to a7d41a8 Compare July 29, 2026 18:48

vlad-scherbich marked this pull request as ready for review July 29, 2026 18:49

vlad-scherbich requested review from a team as code owners July 29, 2026 18:49

vlad-scherbich requested a review from a team as a code owner August 9, 2026 21:15

vlad-scherbich requested a review from rachelyangdog August 9, 2026 21:15

vlad-scherbich force-pushed the vlad/native-heap-gotter-cdylib-build-pr-a branch from 6f9a607 to 5375271 Compare August 9, 2026 21:17

gh-worker-ownership-write-b05516 Bot removed the request for review from a team August 9, 2026 21:17

emmettbutler approved these changes Aug 11, 2026

View reviewed changes

Comment thread setup.py

vlad-scherbich added 17 commits August 12, 2026 16:16

bypass-demo

d5208fc

bot comments

f9982f2

clean up comments

6c9a4ea

fix(profiling): strip heap-gotter cdylib and save debug symbols

e5eba51

Match the linux wheel CI pattern used for _native: extract .debug sidecars with objcopy/strip at staging time so dogfood wheels ship a stripped libdd_heap_gotter.so, while CI debug-symbols.zip still collects the sidecar.

fix(profiling): exclude gotter cdylib from auditwheel repair

90e0ceb

auditwheel iter_versions crashes on the opt-in heap-gotter cdylib ELF when DD_PROFILING_NATIVE_HEAP_BUILD=1, failing build-linux wheel jobs. Exclude the gotter artifact from repair, matching extract_debug_symbols ignores.

Rely on proper native-heap crates in infra

eb06761

ci: retrigger changelog validation after PR title rename

db22c5c

Rename to build(profiling) so changelog/no-changelog is allowed for this inert cdylib packaging change (no customer-facing behavior yet).

Add release note for native heap-gotter cdylib build.

ab3726a

Required for feat PR changelog validation after removing no-changelog label.

ci: drop empty reno stub; use changelog/no-changelog label

47e1d02

Build-only native heap-gotter cdylib wiring has no customer-facing release note.

update Cargo.toml comments

54ca4a5

fix up local ffi; type-annotate

26c92f5

add TODO for integrating heap-gotter into native workspace as a Cargo…

71f235f

… feature

fix(profiling): stash heap-gotter cdylib out of wheel across auditwhe…

a7e3bbb

…el repair

vlad-scherbich force-pushed the vlad/native-heap-gotter-cdylib-build-pr-a branch from 484088e to a7e3bbb Compare August 12, 2026 13:17

vlad-scherbich changed the title ~~build(profiling): add native heap-gotter cdylib build (PROF-15423)~~ build(profiling): add native heap-gotter cdylib build (PROF-15750) Aug 12, 2026

gh-worker-dd-mergequeue-cf854d Bot merged commit 5a832dc into main Aug 12, 2026
1308 checks passed

gh-worker-dd-mergequeue-cf854d Bot deleted the vlad/native-heap-gotter-cdylib-build-pr-a branch August 12, 2026 15:37

KowalskiThomas reviewed Aug 13, 2026

View reviewed changes

Comment thread .gitlab/scripts/build-wheel-helpers.sh

Uh oh!

Conversation

vlad-scherbich commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Testing

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Circular import analysis

⚠️ Existing circular imports

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codeowners resolved as

Uh oh!

datadog-datadog-prod-us1-2 Bot commented Jul 15, 2026 • edited by datadog-prod-us1-4 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pr-commenter Bot commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

scenario:httppropagationextract-datadog_tracecontext_tracestate_not_propagated_on_trace_id_no_match

scenario:httppropagationinject-ids_only

scenario:iastaspects-lstrip_aspect

scenario:iastaspects-translate_aspect

scenario:iastaspectsospath-ospathbasename_aspect

scenario:span-start

scenario:telemetryaddmetric-1-count-metric-1-times

scenario:telemetryaddmetric-record-1000-metrics

Uh oh!

vlad-scherbich commented Jul 28, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vlad-scherbich commented Jul 29, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

emmettbutler left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vlad-scherbich commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cit-pr-commenter-54b7da Bot commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dependency direction analysis

⚠️ Existing dependency direction violations

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

vlad-scherbich commented Jul 15, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jul 15, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jul 15, 2026 •

edited

Loading

datadog-datadog-prod-us1-2 Bot commented Jul 15, 2026 •

edited by datadog-prod-us1-4 Bot

Loading

pr-commenter Bot commented Jul 15, 2026 •

edited

Loading

vlad-scherbich commented Aug 12, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Aug 12, 2026 •

edited

Loading