[python_cpu_sleep_gevent_3.12] add CPU/sleep attribution scenario (PROF-14213) - #130
Open
taegyunkim wants to merge 1 commit into
Open
[python_cpu_sleep_gevent_3.12] add CPU/sleep attribution scenario (PROF-14213)#130taegyunkim wants to merge 1 commit into
taegyunkim wants to merge 1 commit into
Conversation
…OF-14213) New scenario that alternates 10 ms CPU bursts with 50 ms sleeps in gevent mode with 4 concurrent staggered greenlets, asserting on (a) total CPU consumed (667M ns/wall-sec, ±25%) and (b) cpu_burst self-attribution as % of cpu-time profile (80% ±10%). Both bounds together act as a regression check for the gevent over-count bug fixed by dd-trace-py PR #18222 (Finding 3 in the python-cpu-accuracy survey): if the bug recurs, total CPU would inflate to ~1.07 CPU-sec/wall-sec, well outside the ±25% margin. Adaptive sampling is disabled (_DD_PROFILING_STACK_ADAPTIVE_SAMPLING_ENABLED=0). Also adds base_images/Dockerfile.python-3.12.
|
KowalskiThomas
approved these changes
Jun 2, 2026
KowalskiThomas
left a comment
KowalskiThomas
left a comment
Contributor
There was a problem hiding this comment.
Makes sense to me.
Contributor
There was a problem hiding this comment.
Shouldn't this also check that each Task is attributed the right amount of CPU time?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This scenario regression-tests the fix in dd-trace-py PR #18222 (
fix(profiling): swap on-CPU greenlet to position 0 in unwind_greenlets). That fix is merged intodd-trace-py/mainbut is not in any released ddtrace tag (latest release is4.9.0; the fix landed after4.10.0rc1was tagged). Until4.10.0ships to PyPI, this scenario will fail in CI by ~1.6× total-CPU over-count.Do not merge this PR until
ddtrace==4.10.0(or any release that contains PR #18222) is on PyPI — merging earlier will break the prof-correctness CI on main and page #profiling-library-pager on every scheduled run.Track via
pip index versions ddtrace --preor by checking thatgit tag --contains 0c96beab64(in dd-trace-py) lists a release tag.Summary
python_cpu_sleep_gevent_3.12exercising CPU + sleep alternation in gevent mode (M=4 staggered greenlets) on Python 3.12.cpu_burstself-attribution (80%, ±10%).base_images/Dockerfile.python-3.12(also added by the sibling sync + asyncio PRs; rebase whichever lands second/third).Why Python 3.12
Python 3.12 is the most widely deployed version among dd-trace-py customers and the version Datadog runs internally, so the regression check exercises the same combination most users see in production.
Why these bounds
Based on the python-cpu-accuracy survey (DataDog/experimental, users/taegyun.kim/python-cpu-accuracy) adaptive-off gevent run after PR #18222. Same formula as asyncio: M × C / (C + S) = 4 × 0.01 / 0.06 = 0.667 CPU-sec/wall-sec, with
_cpu_loopself-time at ~82% in the survey.Current local-run result (pre-4.10.0 release)
Confirms the regression check fires correctly:
.*(total CPU): expected 2.03e+10 ns, actual 32_894_377_000 ns (~62% over) — bug present.*cpu_burst.*(attribution): expected 80%, actual 52% — bug presentThese numbers will swing to ≤5% error on both metrics once
ddtrace==4.10.0is released and CI picks it up.Test plan
TEST_SCENARIOS="python_cpu_sleep_gevent_3.12" go test -timeout 10m -v -run TestScenariospasses locally (currently fails — blocked on 4.10.0)python.*job passes (currently fails — blocked on 4.10.0)ddtrace==4.10.0is on PyPI, verify CI green and re-request reviewJIRA: PROF-14213