feat(profiling): add Linux CPU timer stack profiler - #18724
Conversation
Codeowners resolved asResolved from the full PR diff against |
|
BenchmarksBenchmark execution time: 2026-08-14 17:27:23 Comparing candidate commit 5c01fee in PR branch Found 0 performance improvements and 5 performance regressions! Performance is the same for 616 metrics, 10 unstable metrics.
|
|
@taegyunkim do you need help reviewing on this PR ? |
taegyunkim
left a comment
There was a problem hiding this comment.
@r1viollet I'd want to make more changes to this PR. Will let you know when review is needed!
Circular import analysis
|
351ca0e to
3ebcfb0
Compare
## Description Pure refactor of existing stack sampler internals, with no intended behavior or internal contract changes, in preparation for timer_create based cpu time profiling in #18724 Specifically: - Extracts the existing task/greenlet/thread-stack rendering branch in `ThreadInfo::sample()` into a local `render_unwound_stacks()` helper. - Extracts duplicate one-time thread registration failure logging in `Sampler::register_thread()` into a local helper. This does not introduce CPU timer profiling, new configuration, new native bindings, or changes to thread registration semantics. ## Testing - `scripts/lint cformat` Attempted a targeted profiling test run from the worktree, but the test runner failed during riot venv setup before executing tests due the worktree using `/home/bits/project/.riot`, which was not writable/available in that context. ## Risks Low. This is intended to be behavior-preserving refactoring only. ## Additional Notes No changelog needed, internal refactor only. Co-authored-by: taegyun.kim <taegyun.kim@datadoghq.com>
Explain why ThreadAltStack adopts a pre-existing alternate signal stack instead of replacing it: the application, CPython's faulthandler, or libdatadog's crashtracker (which by default creates and owns a larger alt stack) may have installed it, and the destructor must never disable or free an alt stack this object did not allocate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a native gtest for ThreadAltStack: an adopted (pre-existing) alternate signal stack must not be disabled on thread-local cleanup, while a stack this helper allocated is disabled and freed. The adopted-stack case fails without the owns_mapping guard, so it locks in this PR's behavior. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…d it ThreadAltStack's destructor disabled whatever alternate signal stack was installed on the thread whenever it owned a mapping, without checking that the installed stack was still the one it installed. If another component (for example libdatadog crashtracker) replaced the thread's alt stack after we installed ours, teardown would strip that replacement, degrading the other owner's fault handling. Only disable the alt stack when it is still ours (cur.ss_sp == mem); free our own mapping regardless. Adds a regression test (DestructorDoesNotDisableReplacedAltStack) that fails without this guard. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Description The profiling safe-copy helper uses a per-thread alternate signal stack for its `SIGSEGV` and `SIGBUS` handlers. When an alternate stack was already installed by the application, CPython's faulthandler, or crashtracker, `ThreadAltStack` reused it but still disabled it during thread-local teardown. That could leave the original owner without its alternate stack and weaken its fault handling. Track whether `ThreadAltStack` allocated the mapping itself. Teardown now disables and unmaps only profiler-owned stacks, while adopted stacks remain installed for their original owner. The installation failure path also unmaps a profiler allocation when `sigaltstack()` fails. A release note documents the fault-handling fix. ## Testing - Passed locally: `ddtrace/internal/datadog/profiling/build_standalone.sh -- RelWithDebInfo stack_test` - 4/4 native CTests passed - verifies that an adopted alternate stack remains installed after `ThreadAltStack` teardown - verifies that a profiler-owned alternate stack is disabled during teardown - `scripts/lint cformat` ## Risks Low. The change narrows teardown to resources owned by the profiler and does not alter public APIs or configuration. Profiler-owned alternate stacks retain their existing cleanup behavior. ## Additional Notes Stacked out from the CPU timer profiler work, #18724, to keep this native safety fix independently reviewable. Follow-up PR #19026 handles the complementary case where another component replaces a profiler-owned alternate stack after the profiler installs it. [PROF-14213](https://datadoghq.atlassian.net/browse/PROF-14213) [PROF-14213]: https://datadoghq.atlassian.net/browse/PROF-14213?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ Co-authored-by: taegyun.kim <taegyun.kim@datadoghq.com>
## Description Strengthens the existing uWSGI `--lazy-apps` test without `--master` by checking the PID returned from `waitpid()` and rejecting signal-based termination. This closes a test blind spot where the process could crash during shutdown while the test continued to validate generated profile samples. The sibling worker is cleaned up before the captured exit status is asserted, including on the failure path. The change is extracted from #18724 because it is independent of the CPU timer profiler. ## Testing - `scripts/run-tests --venv 1ef9287 -- -s -- -k test_uwsgi_threads_processes_no_primary_lazy_apps` - `1 passed, 12 deselected` on Python 3.13.13 with `uwsgi<2.0.30` - `scripts/lint fmt -- tests/profiling/test_uwsgi.py` - `scripts/lint checks` - `git diff --check` ## Risks None. This only makes an existing test detect process crashes that it previously overlooked. ## Additional Notes No release note is needed because this is a test-only change. Co-authored-by: taegyun.kim <taegyun.kim@datadoghq.com>
## Description Strengthens the existing uWSGI `--lazy-apps` test without `--master` by checking the PID returned from `waitpid()` and rejecting signal-based termination. This closes a test blind spot where the process could crash during shutdown while the test continued to validate generated profile samples. The sibling worker is cleaned up before the captured exit status is asserted, including on the failure path. The change is extracted from #18724 because it is independent of the CPU timer profiler. ## Testing - `scripts/run-tests --venv 1ef9287 -- -s -- -k test_uwsgi_threads_processes_no_primary_lazy_apps` - `1 passed, 12 deselected` on Python 3.13.13 with `uwsgi<2.0.30` - `scripts/lint fmt -- tests/profiling/test_uwsgi.py` - `scripts/lint checks` - `git diff --check` ## Risks None. This only makes an existing test detect process crashes that it previously overlooked. ## Additional Notes No release note is needed because this is a test-only change. Co-authored-by: taegyun.kim <taegyun.kim@datadoghq.com>
Dependency direction analysis
|
ca1a6da to
e276144
Compare
e276144 to
1847103
Compare
## Description Consolidates the two native frame updates performed for each greenlet switch into one private `record_greenlet_switch()` call and one `greenlet_info_map` mutex acquisition. The existing behavior is preserved: the origin frame is always updated, while the running target frame is left unchanged when it is needed for parent-greenlet stack unwinding. This is an independently mergeable prerequisite extracted from #18724. ## Testing - `scripts/lint fmt -- ddtrace/profiling/_gevent.py tests/profiling/test_gevent.py tests/profiling/collector/test_stack.py` - `scripts/lint cformat` - `scripts/lint profiling-native-check` - `scripts/lint checks` - `git diff --check` - Fresh CPython 3.12 native extension build - Native linked-parent switch regression covering preserved and updated target frames - Existing high-cardinality greenlet switch contention regression ### Benchmark The benchmark isolates the native operation changed by this PR. Each measured run performs 1,000,000 logical greenlet switch updates after 10,000 warm-up updates: - baseline `e11d640110`: two `update_greenlet_frame()` native calls and two mutex acquisitions per update - candidate `93702898f8`: one `record_greenlet_switch()` native call and one mutex acquisition per update Both `_stack` extensions were built in Release mode from the exact commits, then run in the same container pinned to CPU 0 with `--cpuset-cpus=0`. Results over five runs: - before: 0.531681 seconds median, runs `[0.528339, 0.531681, 0.532128, 0.534566, 0.529234]` - after: 0.366206 seconds median, runs `[0.380415, 0.368066, 0.360803, 0.362898, 0.366206]` - change: approximately 31% faster for the isolated native update Environment: - AWS KVM VM, Intel Xeon Platinum 8175M at 2.50 GHz - 16 vCPUs, 8 cores with 2 threads per core, 61 GiB RAM - Linux 6.8.0-1055-aws, x86-64 - Docker server 29.5.2, container pinned to one CPU - CPython 3.12.13, GCC 14.2.0, glibc 2.41 - container image `sha256:ce7c46dbb7f07d352aecd756e19fd7f39550a72d4a13e68d30f02f5448eebd69` <details> <summary>Benchmark script</summary> ```python import statistics import time from ddtrace.internal.datadog.profiling import stack ITERATIONS = 1_000_000 RUNS = 5 ORIGIN_ID = 101 TARGET_ID = 102 stack.track_greenlet(ORIGIN_ID, "origin", False) stack.track_greenlet(TARGET_ID, "target", False) if hasattr(stack, "record_greenlet_switch"): def update(): stack.record_greenlet_switch(ORIGIN_ID, False, TARGET_ID, None, True) else: def update(): stack.update_greenlet_frame(ORIGIN_ID, False) stack.update_greenlet_frame(TARGET_ID, None) def run_once(): start = time.perf_counter() for _ in range(ITERATIONS): update() return time.perf_counter() - start for _ in range(10_000): update() runs = [run_once() for _ in range(RUNS)] stack.untrack_greenlet(ORIGIN_ID) stack.untrack_greenlet(TARGET_ID) print(f"median={statistics.median(runs):.6f}s runs={runs}") ``` </details> This is an isolated native-call benchmark, not an end-to-end application throughput claim. The earlier 67% figure was discarded after reproducing with exact baseline and candidate builds because the candidate run had not activated the profiling hook. ## Risks Low. The changed Python and native APIs are private and ship together. Existing parent-greenlet frame retention remains unchanged. ## Additional Notes The GIL release around native state mutation is retained intentionally. It was introduced by #14852 in commit `54a3a0ea35` to prevent potential thread-pool deadlocks while waiting for profiler mutexes. This change reduces two such release windows per greenlet switch to one. No release note is needed because this is an internal performance optimization with no user-facing API or behavior change. Co-authored-by: taegyun.kim <taegyun.kim@datadoghq.com>
…-timer-create # Conflicts: # ddtrace/internal/datadog/profiling/stack/src/sampler.cpp # ddtrace/profiling/_gevent.py
This reverts commit c14d1fa.
This reverts commit 7e736ea.
…-timer-create # Conflicts: # ddtrace/internal/datadog/profiling/stack/__init__.pyi # ddtrace/internal/datadog/profiling/stack/_stack.pyi # ddtrace/internal/datadog/profiling/stack/echion/echion/threads.h # ddtrace/internal/datadog/profiling/stack/include/stack_renderer.hpp # ddtrace/internal/datadog/profiling/stack/src/stack.cpp # ddtrace/internal/datadog/profiling/stack/src/stack_renderer.cpp # ddtrace/profiling/collector/stack.py
c81693b to
5c01fee
Compare
Note
Thorough manual review is still in progress.
Description
Adds an opt-in Linux CPU timer path for the stack profiler on GIL-enabled CPython 3.12+. Enable it with
_DD_PROFILING_STACK_CPU_TIMER_ENABLED=true; the default interval is 10 ms and the feature remains off by default.Each Python thread receives a
timer_createper-thread CPU timer that deliversSIGPROFthroughSIGEV_THREAD_ID. The signal handler validates that the signal came from this profiler, captures bounded raw frame and task identity data into a preallocated per-thread SPSC ring, and returns without allocating or locking. The sampler thread drains the rings, validates copied CPython data, and emitscpu-timesamples with asyncio task or greenlet ancestry when the captured identity can be matched safely.Wall sampling continues independently. When CPU timer mode is configured, wall samples stop reporting CPU time so the process does not mix accounting methods if the timer later disables itself.
Safety and lifecycle handling includes:
SIGPROFsignals to the previous handlerPROF-14213
Testing
Coverage includes:
ppoll,read/readv, andnanosleepvariantsscripts/lint cformat,scripts/lint profiling-native-check,scripts/lint checks, andgit diff --checkOn a CPython 3.12 workload with 501 asyncio tasks, selective task reconstruction reduced median CPU timer drain cost from about 2.82 seconds to 0.249 seconds, a 91% reduction, while preserving attribution and wall-sample recovery.
Risks
This path depends on asynchronous signals and version-specific CPython frame layouts. It is Linux-only, private, opt-in, and disabled by default. Unsupported or unsafe runtime conditions disable CPU timer sampling rather than falling back to mixed CPU accounting.
A pending
SIGPROFcan interrupt a native extension syscall that neither masks the signal nor retriesEINTR. CPython's PEP 475 paths retry affected syscalls, and the test suite documents the remaining rawppolllimitation with an expected-failure reproducer.Additional Notes
#19428 now bounds the number of leaf task and greenlet samples emitted by each wall-sampling cycle. CPU timer samples remain outside that reservoir because each signal sample matches at most one captured task or greenlet and emits one CPU sample.
PRs spun out from this work
Span-to-profile correlation is covered by the separate span-attribution chain spun out from this work. It keeps correlation working for physical thread stacks, asyncio task stacks, and greenlet stacks:
Prerequisites already merged into
main:record_greenlet_switch()APIOpen CI prerequisite:
Open task-discovery prerequisite:
Open max-frame prerequisite chain:
Independent test hardening already merged into
main: