feat(profiling): add Linux CPU timer stack profiler

[taegyunkim]

Note

Thorough manual review is still in progress.

Description

Adds an opt-in Linux CPU timer path for the stack profiler on GIL-enabled CPython 3.12+. Enable it with _DD_PROFILING_STACK_CPU_TIMER_ENABLED=true; the default interval is 10 ms and the feature remains off by default.

Each Python thread receives a timer_create per-thread CPU timer that delivers SIGPROF through SIGEV_THREAD_ID. The signal handler validates that the signal came from this profiler, captures bounded raw frame and task identity data into a preallocated per-thread SPSC ring, and returns without allocating or locking. The sampler thread drains the rings, validates copied CPython data, and emits cpu-time samples with asyncio task or greenlet ancestry when the captured identity can be matched safely.

Wall sampling continues independently. When CPU timer mode is configured, wall samples stop reporting CPU time so the process does not mix accounting methods if the timer later disables itself.

Safety and lifecycle handling includes:

• forwarding foreign SIGPROF signals to the previous handler
• guarded CPython memory reads through the existing fault-recovery mechanism
• permanent disable on incompatible signal masks, handler replacement, repeated capture failures, or unsupported platforms
• thread discovery, thread churn, fork, shutdown, and one-shot restart handling
• private diagnostics for dropped samples, timer overruns, capture failures, and disable reasons

PROF-14213

Testing

Coverage includes:

• native tests for the SPSC sample ring and lazy lock-free TID table
• integration tests for CPU sample emission, thread discovery/churn, fork/restart, signal ownership, handler replacement, and fault recovery
• asyncio attribution and parent-task stitching across CPython 3.12, 3.13, and 3.14
• greenlet attribution and logical parent stitching
• syscall hazard reproducers for ppoll, read/readv, and nanosleep variants
• the profiling suite with CPU timer sampling enabled under the default fast-copy configuration, plus focused fast-copy-disabled coverage
• scripts/lint cformat, scripts/lint profiling-native-check, scripts/lint checks, and git diff --check
• latest-main validation with CPython 3.10 and 3.12 native builds, linked-parent greenlet tests, CPU timer ring/TID-table tests, and CPU timer gevent attribution
• post-feat(profiling): reservoir sampling for tasks #19428 validation with a complete CPython 3.10 native build and native sampling-cycle, ring, and TID-table tests
• task-visitor validation with CPython 3.10 and 3.14 native builds and hundreds of correctly attributed, parent-stitched Python 3.14 asyncio CPU samples across repeated checks

On a CPython 3.12 workload with 501 asyncio tasks, selective task reconstruction reduced median CPU timer drain cost from about 2.82 seconds to 0.249 seconds, a 91% reduction, while preserving attribution and wall-sample recovery.

Risks

This path depends on asynchronous signals and version-specific CPython frame layouts. It is Linux-only, private, opt-in, and disabled by default. Unsupported or unsafe runtime conditions disable CPU timer sampling rather than falling back to mixed CPU accounting.

A pending SIGPROF can interrupt a native extension syscall that neither masks the signal nor retries EINTR. CPython's PEP 475 paths retry affected syscalls, and the test suite documents the remaining raw ppoll limitation with an expected-failure reproducer.

Additional Notes

#19428 now bounds the number of leaf task and greenlet samples emitted by each wall-sampling cycle. CPU timer samples remain outside that reservoir because each signal sample matches at most one captured task or greenlet and emits one CPU sample.

PRs spun out from this work

Span-to-profile correlation is covered by the separate span-attribution chain spun out from this work. It keeps correlation working for physical thread stacks, asyncio task stacks, and greenlet stacks:

• fix(profiling): clear finished physical span links #19529, clears finished physical span links
• fix(profiling): isolate gevent span links #19420, adds the shared logical span-link helpers
• fix(profiling): attribute asyncio task samples to inherited spans #19346, attributes asyncio task samples to inherited spans
• fix(profiling): isolate gevent span links #19379, isolates greenlet span links

Prerequisites already merged into main:

• chore(profiling): fix native coverage check #18973, fixes and enables the profiling native coverage check
• refactor(profiling): split stack sampler helpers #18975, extracts stack sampler rendering and registration helpers
• fix(profiling): track signal altstack ownership #19018, tracks alternate signal stack ownership
• fix(profiling): don't disable an alt stack replaced after we installed it #19026, preserves an alternate signal stack installed by another component
• fix(profiling): align frame limit with backend location cap #19076, aligns the frame limit with the backend location cap
• perf(profiling): consolidate greenlet switch updates #19655, consolidates greenlet switch recording through the private record_greenlet_switch() API

Open CI prerequisite:

• ci(profiling): remove duplicate fast-copy profile runs #19684, removes redundant explicit fast-copy full-suite runs now that fast copy is enabled by default

Open task-discovery prerequisite:

• refactor(profiling): separate task address discovery #19709, shares task-source traversal through an address visitor while preserving immediate wall-task snapshots

Open max-frame prerequisite chain:

• perf(profiling): honor configured maximum frame count #19073, propagates the configured maximum frame count through native stack collection
• perf(profiling): bound task stack collection #19085, stacked on perf(profiling): honor configured maximum frame count #19073, bounds task-stack collection within the configured frame budget

Independent test hardening already merged into main:

• test(profiling): detect uWSGI worker crashes #19405, makes the uWSGI regression detect worker crashes

[cit-pr-commenter-54b7da]

Codeowners resolved as

Resolved from the full PR diff against main using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

ddtrace/internal/datadog/profiling/dd_wrapper/include/profiler_stats.hpp  @DataDog/profiling-python
ddtrace/internal/datadog/profiling/dd_wrapper/src/profiler_stats.cpp    @DataDog/profiling-python
ddtrace/internal/datadog/profiling/stack/CMakeLists.txt                 @DataDog/profiling-python
ddtrace/internal/datadog/profiling/stack/__init__.py                    @DataDog/profiling-python
ddtrace/internal/datadog/profiling/stack/__init__.pyi                   @DataDog/profiling-python
ddtrace/internal/datadog/profiling/stack/_stack.pyi                     @DataDog/profiling-python
ddtrace/internal/datadog/profiling/stack/echion/echion/danger.h         @DataDog/profiling-python
ddtrace/internal/datadog/profiling/stack/echion/echion/threads.h        @DataDog/profiling-python
ddtrace/internal/datadog/profiling/stack/include/cpu_sample_ring.hpp    @DataDog/profiling-python
ddtrace/internal/datadog/profiling/stack/include/cpu_timer.hpp          @DataDog/profiling-python
ddtrace/internal/datadog/profiling/stack/include/cpu_timer_tid_table.hpp  @DataDog/profiling-python
ddtrace/internal/datadog/profiling/stack/include/sampler.hpp            @DataDog/profiling-python
ddtrace/internal/datadog/profiling/stack/include/stack_renderer.hpp     @DataDog/profiling-python
ddtrace/internal/datadog/profiling/stack/src/cpu_timer.cpp              @DataDog/profiling-python
ddtrace/internal/datadog/profiling/stack/src/echion/danger.cc           @DataDog/profiling-python
ddtrace/internal/datadog/profiling/stack/src/echion/threads.cc          @DataDog/profiling-python
ddtrace/internal/datadog/profiling/stack/src/sampler.cpp                @DataDog/profiling-python
ddtrace/internal/datadog/profiling/stack/src/stack.cpp                  @DataDog/profiling-python
ddtrace/internal/datadog/profiling/stack/src/stack_renderer.cpp         @DataDog/profiling-python
ddtrace/internal/datadog/profiling/stack/test/CMakeLists.txt            @DataDog/profiling-python
ddtrace/internal/datadog/profiling/stack/test/test_cpu_sample_ring.cpp  @DataDog/profiling-python
ddtrace/internal/datadog/profiling/stack/test/test_cpu_timer_tid_table.cpp  @DataDog/profiling-python
ddtrace/internal/settings/profiling.py                                  @DataDog/profiling-python
ddtrace/profiling/collector/stack.py                                    @DataDog/profiling-python
releasenotes/notes/prof-14213-cpu-timer-profiler.yaml                   @DataDog/apm-python
riotfile.py                                                             @DataDog/apm-python
tests/profiling/cpu_timer_native_syscall_hazard_app.py                  @DataDog/profiling-python
tests/profiling/native_cpu_timer_syscall_hazards.c                      @DataDog/profiling-python
tests/profiling/test_cpu_timer.py                                       @DataDog/profiling-python
tests/profiling/test_cpu_timer_native_syscalls.py                       @DataDog/profiling-python
tests/profiling/test_profiling_config.py                                @DataDog/profiling-python

[datadog-datadog-prod-us1]

 

✨ Unblock PR with BitsAI

⚠️ Warnings

🚦 12 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-py | build linux serverless: [amd64, cp315-cp315, v126532274-233089d-musllinux_1_2_x86_64, 1]     

DataDog/apm-reliability/dd-trace-py | build linux serverless: [arm64, cp315-cp315, v113741357-d2b8243-manylinux2014_aarch64, 1]     

DataDog/apm-reliability/dd-trace-py | build linux: [amd64, cp315-cp315, v113741238-d2b8243-manylinux2014_x86_64]     

View all 12 failed jobs.

❄️ 1 New flaky test detected

    test_cpu_timer_disables_when_fault_handler_is_replaced from test_cpu_timer.py  

View in Flaky Test Management

ℹ️ Info

No other issues found (see more)

🧪 All tests passed

🔄 Datadog auto-retried 3 jobs - 0 passed on retry

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 5c01fee | Docs | Datadog PR Page | Give us feedback!}

[pr-commenter]

Benchmarks

Benchmark execution time: 2026-08-14 17:27:23

Comparing candidate commit 5c01fee in PR branch taegyunkim/prof-14213-timer-create with baseline commit 6f2816b in branch main.

📊 Benchmarking dashboard

Found 0 performance improvements and 5 performance regressions! Performance is the same for 616 metrics, 10 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

• 🟩 = significantly better candidate vs. baseline
• 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:httppropagationinject-ids_only

• 🟥 execution_time [+2.448µs; +2.668µs] or [+11.301%; +12.314%]

scenario:iastaspectsospath-ospathbasename_aspect

• 🟥 execution_time [+114.703µs; +124.155µs] or [+27.074%; +29.305%]

scenario:span-start

• 🟥 execution_time [+1.220ms; +1.385ms] or [+8.024%; +9.113%]

scenario:telemetryaddmetric-1-count-metric-1-times

• 🟥 execution_time [+480.436ns; +528.839ns] or [+18.095%; +19.918%]

scenario:tracer-small

• 🟥 execution_time [+32.460µs; +34.553µs] or [+9.956%; +10.598%]

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:coreapiscenario-context_with_data_listeners

• unstable execution_time [-767.439ns; +710.737ns] or [-7.019%; +6.500%]

scenario:coreapiscenario-core_dispatch_1_listener

• unstable execution_time [-34.657ns; +31.018ns] or [-5.691%; +5.093%]

scenario:coreapiscenario-core_dispatch_50_listeners

• unstable execution_time [-1617.100ns; +1661.450ns] or [-9.537%; +9.799%]

scenario:coreapiscenario-core_dispatch_exception_listeners

• unstable execution_time [-1312.228ns; +1165.895ns] or [-10.106%; +8.979%]

scenario:coreapiscenario-core_dispatch_listeners

• unstable execution_time [-322.291ns; +326.060ns] or [-8.779%; +8.882%]

scenario:coreapiscenario-core_dispatch_no_args_listeners

• unstable execution_time [-255.693ns; +254.726ns] or [-8.738%; +8.705%]

scenario:coreapiscenario-core_dispatch_with_results_1_listener

• unstable execution_time [-75.048ns; +72.103ns] or [-6.486%; +6.231%]

scenario:coreapiscenario-core_dispatch_with_results_50_listeners

• unstable execution_time [-4004.318ns; +4075.020ns] or [-9.781%; +9.953%]

scenario:coreapiscenario-core_dispatch_with_results_listeners

• unstable execution_time [-855.364ns; +692.193ns] or [-10.487%; +8.487%]

scenario:packagesupdateimporteddependencies-import_many_stdlib_cached

• unstable execution_time [-42952.781ns; +41830.580ns] or [-6.729%; +6.553%]

[r1viollet]

@taegyunkim do you need help reviewing on this PR ?

[taegyunkim]

@r1viollet I'd want to make more changes to this PR. Will let you know when review is needed!

[cit-pr-commenter-54b7da]

Circular import analysis

⚠️ Existing circular imports

There are 5 circular imports that already exist on the base branch and have not been changed by this PR.

ddtrace.contrib.internal.django.patch -> ddtrace.contrib.internal.django.response -> ddtrace.contrib.internal.django.patch
ddtrace.contrib.internal.pytorch._distributed -> ddtrace.contrib.internal.pytorch._rank_root -> ddtrace.contrib.internal.pytorch._distributed
ddtrace.llmobs -> ddtrace.llmobs._evaluators -> ddtrace.llmobs._evaluators.format -> ddtrace.llmobs._experiment -> ddtrace.llmobs
ddtrace.errortracking._handled_exceptions.bytecode_injector -> ddtrace.errortracking._handled_exceptions.callbacks -> ddtrace.errortracking._handled_exceptions.collector -> ddtrace.errortracking._handled_exceptions.bytecode_reporting -> ddtrace.errortracking._handled_exceptions.bytecode_injector
ddtrace.appsec._asm_request_context -> ddtrace.appsec._iast._iast_request_context_base -> ddtrace.appsec._iast._iast_env -> ddtrace.appsec._iast.reporter -> ddtrace.appsec._exploit_prevention.stack_traces -> ddtrace.appsec._asm_request_context

[cit-pr-commenter-54b7da]

Dependency direction analysis

⚠️ Existing dependency direction violations

There are 255 dependency direction violations that already exist on the base branch and have not been changed by this PR.

Show existing violations (showing 5 of 255 highest severity)

ddtrace.internal.tracemethods -×-> ddtrace.trace  (internal-core -> product:tracing, score=134)
ddtrace.llmobs._integrations.claude_agent_sdk -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=132)
ddtrace.llmobs._evaluators.runner -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=132)
ddtrace.llmobs._integrations.google_adk -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=132)
ddtrace.internal.openfeature._span_enrichment -×-> ddtrace.trace  (product:openfeature -> product:tracing, score=132)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:

uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json