diff --git a/README.md b/README.md index 93c6140..54908d8 100644 --- a/README.md +++ b/README.md @@ -77,7 +77,7 @@ Both use [dd-octo-sts](https://github.com/DataDog/dd-octo-sts-action) (`dd-trace **Inputs:** - `dd_trace_py_commit_sha` — commit to test (required) -- `test_scenarios` — regexp passed to `TEST_SCENARIOS` (dd-trace-py passes the [6-scenario 3.14/3.15 migration gate](scenarios/python_downstream_gate/README.md); the downstream workflow default alone is `python.*`) +- `test_scenarios` — regexp passed to `TEST_SCENARIOS` (dd-trace-py passes the [8-scenario 3.14/3.15 migration gate](scenarios/python_downstream_gate/README.md); the downstream workflow default alone is `python.*`) ## Creating new tests diff --git a/scenarios/python_cpu_3.14/Dockerfile b/scenarios/python_cpu_3.14/Dockerfile new file mode 100644 index 0000000..b0ba06e --- /dev/null +++ b/scenarios/python_cpu_3.14/Dockerfile @@ -0,0 +1,15 @@ +ARG BASE_IMAGE="prof-python-3.14" +FROM $BASE_IMAGE + +COPY ./scenarios/python_cpu_3.14/requirements.txt /app/requirements.txt +RUN pip install -r /app/requirements.txt + +COPY ./scenarios/python_cpu_3.14/main.py /app/main.py +WORKDIR /app + +ENV EXECUTION_TIME_SEC=10 +ENV _DD_PROFILING_STACK_ADAPTIVE_SAMPLING_ENABLED=0 +ENV DD_PROFILING_ENABLED=true +ENV DD_PROFILING_MEMORY_ENABLED=false + +CMD ddtrace-run python main.py diff --git a/scenarios/python_cpu_3.14/README.md b/scenarios/python_cpu_3.14/README.md new file mode 100644 index 0000000..cb81083 --- /dev/null +++ b/scenarios/python_cpu_3.14/README.md @@ -0,0 +1,24 @@ +## CPU stack profiling (3.14 baseline) + +Validates that the Datadog Python profiler's **stack collector** reports +`cpu-time` samples with correct call stacks and `thread name` labels under a +CPU-bound workload. This is the **3.14 baseline** half of the `3.14 -> 3.15` +migration pair (see `python_cpu_3.15`). + +Reuses the workload from `scenarios/python_cpu`: two tight loops (`a` and `b`) +with a 2:1 relative CPU share. Memory profiling is disabled so the assertion +targets stack samples only. + +## Expected profile + +- `cpu-time`: + - `;.*main;.*b` ~= 66% (`MainThread`) + - `;.*main;.*a` ~= 33% (`MainThread`) + +`cpu-time` is best-effort per sample, so `error_margin` is generous. +`scale_by_duration` normalizes across run lengths. + +## Notes + +The 3.14 scenario runs on prof-correctness `main` CI (PyPI ddtrace). The 3.15 +candidate runs only via the dd-trace-py downstream gate (`DDTRACE_INSTALL_URL`). diff --git a/scenarios/python_cpu_3.14/expected_profile.json b/scenarios/python_cpu_3.14/expected_profile.json new file mode 100644 index 0000000..607d1b9 --- /dev/null +++ b/scenarios/python_cpu_3.14/expected_profile.json @@ -0,0 +1,37 @@ +{ + "test_name": "python_cpu_3.14", + "stacks": [ + { + "profile-type": "cpu-time", + "stack-content": [ + { + "regular_expression": ";.*main;.*b", + "percent": 99, + "error_margin": 0, + "labels": [ + { + "key": "thread name", + "values": [ + "MainThread" + ] + } + ] + }, + { + "regular_expression": ";.*main;.*a", + "percent": 33, + "error_margin": 100, + "labels": [ + { + "key": "thread name", + "values": [ + "MainThread" + ] + } + ] + } + ] + } + ], + "scale_by_duration": true +} diff --git a/scenarios/python_cpu_3.14/main.py b/scenarios/python_cpu_3.14/main.py new file mode 100644 index 0000000..f7d850f --- /dev/null +++ b/scenarios/python_cpu_3.14/main.py @@ -0,0 +1,30 @@ +import os +from time import time + + +class CPUBurner: + def __init__(self) -> None: + self.x = 0 + self.i = 0 + + def a(self) -> None: + self.i = 0 + while self.i < 1000000: + self.x += self.i + self.i += 1 + + def b(self) -> None: + self.i = 0 + while self.i < 2000000: + self.x += self.i + self.i += 1 + + def main(self) -> None: + execution_time_sec = float(os.getenv("EXECUTION_TIME_SEC", "10")) + end = time() + execution_time_sec + while time() < end: + self.a() + self.b() + + +CPUBurner().main() diff --git a/scenarios/python_cpu_3.14/requirements.txt b/scenarios/python_cpu_3.14/requirements.txt new file mode 100644 index 0000000..749bf29 --- /dev/null +++ b/scenarios/python_cpu_3.14/requirements.txt @@ -0,0 +1 @@ +ddtrace diff --git a/scenarios/python_cpu_3.15/Dockerfile b/scenarios/python_cpu_3.15/Dockerfile new file mode 100644 index 0000000..73c3a54 --- /dev/null +++ b/scenarios/python_cpu_3.15/Dockerfile @@ -0,0 +1,16 @@ +ARG BASE_IMAGE="prof-python-3.15" +FROM $BASE_IMAGE + +# ddtrace is pre-installed in the base image when DDTRACE_INSTALL_URL is set +# (dd-trace-py downstream CI). Do not pip-install here — PyPI wheels may not +# exist for 3.15 yet. + +COPY ./scenarios/python_cpu_3.15/main.py /app/main.py +WORKDIR /app + +ENV EXECUTION_TIME_SEC=10 +ENV _DD_PROFILING_STACK_ADAPTIVE_SAMPLING_ENABLED=0 +ENV DD_PROFILING_ENABLED=true +ENV DD_PROFILING_MEMORY_ENABLED=false + +CMD ddtrace-run python main.py diff --git a/scenarios/python_cpu_3.15/README.md b/scenarios/python_cpu_3.15/README.md new file mode 100644 index 0000000..5438756 --- /dev/null +++ b/scenarios/python_cpu_3.15/README.md @@ -0,0 +1,25 @@ +## CPU stack profiling (3.15 candidate) + +Validates that the Datadog Python profiler's **stack collector** reports +`cpu-time` samples with correct call stacks and `thread name` labels under a +CPU-bound workload. This is the **3.15 candidate** half of the `3.14 -> 3.15` +migration pair (see `python_cpu_3.14`). + +Reuses the workload from `scenarios/python_cpu`: two tight loops (`a` and `b`) +with a 2:1 relative CPU share. Memory profiling is disabled so the assertion +targets stack samples only. + +## Expected profile + +- `cpu-time`: + - `;.*main;.*b` ~= 66% (`MainThread`) + - `;.*main;.*a` ~= 33% (`MainThread`) + +`cpu-time` is best-effort per sample, so `error_margin` is generous. +`scale_by_duration` normalizes across run lengths. + +## Notes + +This scenario is wheel-only: PyPI ddtrace wheels may not exist for 3.15 yet, so +it is excluded from prof-correctness `main` CI and runs via the dd-trace-py +downstream gate (`DDTRACE_INSTALL_URL`). diff --git a/scenarios/python_cpu_3.15/expected_profile.json b/scenarios/python_cpu_3.15/expected_profile.json new file mode 100644 index 0000000..f563824 --- /dev/null +++ b/scenarios/python_cpu_3.15/expected_profile.json @@ -0,0 +1,37 @@ +{ + "test_name": "python_cpu_3.15", + "stacks": [ + { + "profile-type": "cpu-time", + "stack-content": [ + { + "regular_expression": ";.*main;.*b", + "percent": 66, + "error_margin": 100, + "labels": [ + { + "key": "thread name", + "values": [ + "MainThread" + ] + } + ] + }, + { + "regular_expression": ";.*main;.*a", + "percent": 33, + "error_margin": 100, + "labels": [ + { + "key": "thread name", + "values": [ + "MainThread" + ] + } + ] + } + ] + } + ], + "scale_by_duration": true +} diff --git a/scenarios/python_cpu_3.15/main.py b/scenarios/python_cpu_3.15/main.py new file mode 100644 index 0000000..f7d850f --- /dev/null +++ b/scenarios/python_cpu_3.15/main.py @@ -0,0 +1,30 @@ +import os +from time import time + + +class CPUBurner: + def __init__(self) -> None: + self.x = 0 + self.i = 0 + + def a(self) -> None: + self.i = 0 + while self.i < 1000000: + self.x += self.i + self.i += 1 + + def b(self) -> None: + self.i = 0 + while self.i < 2000000: + self.x += self.i + self.i += 1 + + def main(self) -> None: + execution_time_sec = float(os.getenv("EXECUTION_TIME_SEC", "10")) + end = time() + execution_time_sec + while time() < end: + self.a() + self.b() + + +CPUBurner().main() diff --git a/scenarios/python_downstream_gate/README.md b/scenarios/python_downstream_gate/README.md index 2a0d622..c661f94 100644 --- a/scenarios/python_downstream_gate/README.md +++ b/scenarios/python_downstream_gate/README.md @@ -1,6 +1,6 @@ # Python downstream gate (dd-trace-py) -Six prof-correctness scenarios exercise the profiling stack on **3.14 +Eight prof-correctness scenarios exercise the profiling stack on **3.14 (baseline)** and **3.15 (candidate)** for the same workloads. They are the default set when dd-trace-py triggers downstream CI on profiling changes. @@ -8,6 +8,7 @@ default set when dd-trace-py triggers downstream CI on profiling changes. | Family | 3.14 (baseline) | 3.15 (candidate) | |--------|-------------------|------------------| +| cpu (stack) | `python_cpu_3.14` | `python_cpu_3.15` | | exceptions | `python_exceptions_3.14` | `python_exceptions_3.15` | | async-gen | `python_async_gen_3.14` | `python_async_gen_3.15` | | lock | `python_lock_3.14` | `python_lock_3.15` | @@ -16,17 +17,18 @@ default set when dd-trace-py triggers downstream CI on profiling changes. | Family | Profile type asserted | Collectors / setup | |--------|----------------------|--------------------| +| cpu (stack) | `cpu-time` + `thread name` | Stack via `ddtrace-run`; memory off; CPU-bound loops | | exceptions | `exception-samples` + `exception type` | Exception profiler | | async-gen | `wall-time` | Full profiler via `ddtrace-run`; asyncio async-generator workload | | lock | `lock-acquire` + `lock-release` + `lock name` | Lock profiler; threaded lock churn | -Feature-specific pairs (mem_domain, live_heap) and extended coverage (cpu, alloc, +Feature-specific pairs (mem_domain, live_heap) and extended coverage (alloc, asyncio, …) land in follow-up PRs. ## Default downstream regexp ``` -python_(exceptions|async_gen|lock)_3\.(14|15) +python_(cpu|exceptions|async_gen|lock)_3\.(14|15) ``` Override via `workflow_dispatch` → `test_scenarios`, or when triggering @@ -41,7 +43,7 @@ Override via `workflow_dispatch` → `test_scenarios`, or when triggering ```sh export DDTRACE_INSTALL_URL="https://dd-trace-py-builds.s3.amazonaws.com//install.sh" -TEST_SCENARIOS='python_(exceptions|async_gen|lock)_3\.(14|15)' go test -v -run TestScenarios +TEST_SCENARIOS='python_(cpu|exceptions|async_gen|lock)_3\.(14|15)' go test -v -run TestScenarios ``` ## Further reading