Skip to content

Commit 36a562e

Browse files
vlad-scherbichvlad-scherbich
committed
feat(python): add live_heap 3.14/3.15 gate scenarios
Persistent live-heap snapshot coverage; both dirs are wheel-only. Add python_live_heap_3.14 to main CI exclude. Stacks on gate-infra (PR-0).
1 parent 2c21efa commit 36a562e

12 files changed

Lines changed: 365 additions & 38 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,7 @@ jobs:
6666
# exclude rather than a lookahead baked into test_scenarios.
6767
# Additional wheel-only 3.14 dirs (e.g. live_heap) are added to this list
6868
# when those scenarios land.
69-
test_scenarios_exclude: '_3\.15$|^python_downstream_gate$'
69+
test_scenarios_exclude: '_3\.15$|^python_live_heap_3\.14$|^python_downstream_gate$'
7070
secrets: inherit
7171
full_host:
7272
uses: ./.github/workflows/test.yml

‎scenarios/python_downstream_gate/README.md‎

Lines changed: 14 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -1,53 +1,30 @@
11
# Python downstream gate (dd-trace-py)
22

3-
Paired **3.14 (baseline)** and **3.15 (candidate)** prof-correctness scenarios
4-
exercise the Python profiling stack for the 3.14 → 3.15 migration. They are the
5-
intended default set when dd-trace-py triggers downstream CI on profiling changes.
6-
7-
**Scenarios land in follow-up PRs** (core families first, then feature-specific
8-
pairs). This directory is an index only — not a runnable scenario.
3+
Two prof-correctness scenarios exercise **persistent live-heap** profiling on
4+
**3.14 (baseline)** and **3.15 (candidate)**. Both dirs are **wheel-only**
5+
(persistent live-heap is not in a published PyPI release yet).
96

107
## Scenarios
118

12-
| Family | 3.14 (baseline) | 3.15 (candidate) | PR |
13-
|--------|-----------------|---------------------|-----|
14-
| _(pending)_ | — | — | Core scenarios in follow-up PRs |
9+
| Family | 3.14 (baseline) | 3.15 (candidate) |
10+
|--------|-------------------|------------------|
11+
| live-heap | `python_live_heap_3.14` | `python_live_heap_3.15` |
1512

1613
## Default downstream regexp
1714

18-
Once scenarios are added, dd-trace-py should pass an explicit regexp (not the
19-
downstream workflow default of `python.*`). The regexp grows as families merge;
20-
see each PR for the current value.
21-
22-
Override via `workflow_dispatch` → `test_scenarios`, or when triggering
23-
[`downstream-python.yml`](../../.github/workflows/downstream-python.yml) manually.
24-
25-
## Wheel install
15+
```
16+
python_live_heap_3\.(14|15)
17+
```
2618

27-
Every scenario builds against a **dd-trace-py wheel** via `DDTRACE_INSTALL_URL`
28-
(as `downstream-python.yml` does:
29-
`https://dd-trace-py-builds.s3.amazonaws.com/<sha>/install.sh`), pre-installed in
30-
the base image.
19+
## CI exclude
3120

32-
- **All `*_3.15` folders** — PyPI wheels may not be published for 3.15 yet;
33-
excluded from prof-correctness `main` CI (see `test_scenarios_exclude` in
34-
[`.github/workflows/ci.yml`](../../.github/workflows/ci.yml)).
35-
- **Wheel-only 3.14 folders** — scenarios that depend on unreleased ddtrace
36-
features are also excluded from `main` CI until the feature ships.
21+
Both folders require `DDTRACE_INSTALL_URL`. `python_live_heap_3.14` is also
22+
listed in [`ci.yml`](../../.github/workflows/ci.yml) `test_scenarios_exclude`
23+
until the feature ships in a PyPI release.
3724

3825
## Local run
3926

4027
```sh
4128
export DDTRACE_INSTALL_URL="https://dd-trace-py-builds.s3.amazonaws.com/<commit-sha>/install.sh"
42-
TEST_SCENARIOS='<gate-regexp>' go test -v -run TestScenarios
29+
TEST_SCENARIOS='python_live_heap_3\.(14|15)' go test -v -run TestScenarios
4330
```
44-
45-
## Gate lifecycle
46-
47-
This gate tests the **migration delta** (3.14 → 3.15). It is time-boxed: retire
48-
the paired 14v15 framing at 3.15 GA and fold workloads into steady-state
49-
prof-correctness on {oldest, newest} supported Python versions.
50-
51-
## Further reading
52-
53-
- prof-correctness downstream wiring: [README](../../README.md#downstream-from-dd-trace-py)

‎scenarios/python_live_heap_3.14/Dockerfile‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
ARG BASE_IMAGE="prof-python-3.14"
2+
FROM $BASE_IMAGE
3+
4+
# ddtrace is pre-installed in the base image when DDTRACE_INSTALL_URL is set
5+
# (dd-trace-py downstream CI). Do not pip-install from PyPI here: the persistent
6+
# live-heap profile is recent and may not be in a published release yet, so this
7+
# scenario runs against the just-built wheel until it ships.
8+
COPY ./scenarios/python_live_heap_3.14/main.py /app/main.py
9+
WORKDIR /app
10+
11+
ENV EXECUTION_TIME_SEC="15"
12+
# Enable the live-heap profiler and make the snapshot the deliverable.
13+
ENV DD_PROFILING_MEMORY_ENABLED=true
14+
ENV DD_PROFILING_HEAP_ENABLED=true
15+
# Sample interval == object size -> ~one sample per object -> stable
16+
# per-stack proportions for both heap-space and heap-live-samples.
17+
ENV DD_PROFILING_HEAP_SAMPLE_SIZE=16384
18+
# Export several snapshots during the run so the persistent live set is
19+
# exercised across upload intervals (not just at shutdown).
20+
ENV DD_PROFILING_UPLOAD_INTERVAL=5
21+
# Isolate the heap profiler so other collectors don't add noise.
22+
ENV DD_PROFILING_STACK_ENABLED=false
23+
ENV DD_PROFILING_LOCK_ENABLED=false
24+
25+
CMD python main.py

‎scenarios/python_live_heap_3.14/README.md‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
## Live-Heap Profiling (3.14 baseline)
2+
3+
Validates that the Datadog Python profiler's **persistent live-heap profile**
4+
correctly reports the set of live (still-allocated) sampled objects. This is the
5+
**3.14 baseline** half of the `3.14 -> 3.15` migration pair (see
6+
`python_live_heap_3.15`).
7+
8+
Unlike `alloc-space`/`alloc-samples` (which count every allocation over the
9+
interval), the live-heap profile is a running snapshot of what is *currently
10+
live*: allocations are added when sampled and subtracted when freed, and the
11+
snapshot is exported non-destructively on every upload. It is attached as a
12+
second pprof (`<prefix>.<pid>.<seq>.heap.pprof`) alongside the primary profile.
13+
14+
## Test Application
15+
16+
`main.py` retains a known live set for the whole run, split into two
17+
distinctly-named call sites that allocate **equal-size** objects (16 KiB) and
18+
differ only in count:
19+
20+
- `retain_major` - 1,600 objects (~25 MiB, ~80% of the live set)
21+
- `retain_minor` - 400 objects (~6 MiB, ~20% of the live set)
22+
23+
Equal sizes mean the 80/20 split holds for both metrics the live-heap profile
24+
reports: `heap-space` (live bytes = count x size) and `heap-live-samples` (live
25+
object count). It uses `bytes` (`PyObject_Malloc`, the OBJ allocator domain) so
26+
the heap profiler tracks the objects identically across versions, independent of
27+
the `DD_PROFILING_MEMORY_MEM_DOMAIN_ENABLED` toggle (`bytearray` moved OBJ -> MEM
28+
in 3.13). The objects are held alive while the process idles through several
29+
upload intervals (`DD_PROFILING_UPLOAD_INTERVAL=5`), so each exported heap
30+
snapshot contains the full live set.
31+
32+
## Expected Profile
33+
34+
Assertions run against the heap snapshot pprof only (selected with
35+
`pprof-regex`). `DD_PROFILING_HEAP_SAMPLE_SIZE=16384` (== object size) yields
36+
roughly one sample per object, keeping the per-stack proportions stable.
37+
38+
- `heap-space` (live bytes):
39+
- `^<module>;Target.run;Target.retain_major$` ~= 80%
40+
- `^<module>;Target.run;Target.retain_minor$` ~= 20%
41+
- `heap-live-samples` (live object count): same ~80/20 split.
42+
43+
`allow_first_profile_failure` tolerates the first snapshot, which may be taken
44+
before the live set is fully built.
45+
46+
## Notes
47+
48+
The persistent live-heap profile is recent, so this scenario runs against a
49+
ddtrace build that includes it (via `DDTRACE_INSTALL_URL` pointing at a
50+
dd-trace-py S3 wheel) rather than a PyPI release. It is excluded from
51+
prof-correctness `main` CI until the feature ships in a release.

‎scenarios/python_live_heap_3.14/expected_profile.json‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
{
2+
"test_name": "python_live_heap",
3+
"note": "Persistent live-heap profile. Assertions run against the separate heap snapshot pprof (<prefix>.<pid>.<seq>.heap.pprof), selected via pprof-regex. heap-space = live bytes, heap-live-samples = live object count. retain_major/retain_minor allocate equal-size objects in an 80/20 count split, so both metrics show the same ~80/20 proportions. Thread-name labels are intentionally not asserted: the heap profiler tags samples with the numeric OS thread id, which varies per run.",
4+
"pprof-regex": ".*heap\\.pprof",
5+
"scale_by_duration": false,
6+
"allow_first_profile_failure": true,
7+
"stacks": [
8+
{
9+
"profile-type": "heap-space",
10+
"stack-content": [
11+
{
12+
"regular_expression": "^<module>;Target\\.run;Target\\.retain_major$",
13+
"percent": 80,
14+
"error_margin": 10,
15+
"labels": []
16+
},
17+
{
18+
"regular_expression": "^<module>;Target\\.run;Target\\.retain_minor$",
19+
"percent": 20,
20+
"error_margin": 10,
21+
"labels": []
22+
}
23+
]
24+
},
25+
{
26+
"profile-type": "heap-live-samples",
27+
"stack-content": [
28+
{
29+
"regular_expression": "^<module>;Target\\.run;Target\\.retain_major$",
30+
"percent": 80,
31+
"error_margin": 12,
32+
"labels": []
33+
},
34+
{
35+
"regular_expression": "^<module>;Target\\.run;Target\\.retain_minor$",
36+
"percent": 20,
37+
"error_margin": 12,
38+
"labels": []
39+
}
40+
]
41+
}
42+
]
43+
}

‎scenarios/python_live_heap_3.14/main.py‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
import os
2+
import time
3+
4+
from ddtrace.profiling import Profiler
5+
6+
# Allocations are held at module scope so they stay live for the whole process
7+
# and therefore appear in every live-heap snapshot the profiler exports.
8+
LIVE: list = []
9+
10+
# Two distinctly-named call sites that retain a known live set. Both allocate
11+
# the SAME object size and differ only in count, so the 80/20 split holds for
12+
# both metrics the live-heap profile reports:
13+
# - heap-space (live bytes) -> count * size -> 80/20
14+
# - heap-live-samples (live objects) -> count -> 80/20
15+
# We use ``bytes`` (PyObject_Malloc / OBJ domain), which the heap profiler
16+
# tracks identically across Python versions and independent of the MEM-domain
17+
# toggle (``bytearray`` moved OBJ -> MEM in 3.13). This keeps the scenario a
18+
# clean live-heap check across the 3.14 -> 3.15 migration.
19+
OBJ_SIZE = 16384 # 16 KiB, well above the pymalloc small-object threshold
20+
N_MAJOR = 1600 # ~80% of the live set (1600 * 16 KiB ~= 25 MiB)
21+
N_MINOR = 400 # ~20% of the live set ( 400 * 16 KiB ~= 6 MiB)
22+
23+
24+
class Target:
25+
def __init__(self) -> None:
26+
self.live: list = []
27+
28+
def run(self, hold_seconds: float) -> None:
29+
self.retain_major()
30+
self.retain_minor()
31+
# Keep the live set alive across several upload intervals so each
32+
# exported heap snapshot contains the full set.
33+
deadline = time.monotonic() + hold_seconds
34+
while time.monotonic() < deadline:
35+
time.sleep(0.5)
36+
37+
def retain_major(self) -> None:
38+
for _ in range(N_MAJOR):
39+
self.live.append(bytes(OBJ_SIZE))
40+
41+
def retain_minor(self) -> None:
42+
for _ in range(N_MINOR):
43+
self.live.append(bytes(OBJ_SIZE))
44+
45+
46+
if __name__ == "__main__":
47+
prof = Profiler()
48+
prof.start() # As early as possible so the allocations below are sampled.
49+
50+
execution_time = int(os.environ.get("EXECUTION_TIME_SEC", "15"))
51+
target = Target()
52+
LIVE.append(target)
53+
target.run(hold_seconds=execution_time)
54+
55+
prof.stop()

‎scenarios/python_live_heap_3.14/requirements.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
ddtrace

‎scenarios/python_live_heap_3.15/Dockerfile‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
ARG BASE_IMAGE="prof-python-3.15"
2+
FROM $BASE_IMAGE
3+
4+
# ddtrace is pre-installed in the base image when DDTRACE_INSTALL_URL is set
5+
# (dd-trace-py downstream CI). Do not pip-install here — PyPI wheels may not
6+
# exist for 3.15 yet, and the persistent live-heap profile is recent.
7+
COPY ./scenarios/python_live_heap_3.15/main.py /app/main.py
8+
WORKDIR /app
9+
10+
ENV EXECUTION_TIME_SEC="15"
11+
# Enable the live-heap profiler and make the snapshot the deliverable.
12+
ENV DD_PROFILING_MEMORY_ENABLED=true
13+
ENV DD_PROFILING_HEAP_ENABLED=true
14+
# Sample interval == object size -> ~one sample per object -> stable
15+
# per-stack proportions for both heap-space and heap-live-samples.
16+
ENV DD_PROFILING_HEAP_SAMPLE_SIZE=16384
17+
# Export several snapshots during the run so the persistent live set is
18+
# exercised across upload intervals (not just at shutdown).
19+
ENV DD_PROFILING_UPLOAD_INTERVAL=5
20+
# Isolate the heap profiler so other collectors don't add noise.
21+
ENV DD_PROFILING_STACK_ENABLED=false
22+
ENV DD_PROFILING_LOCK_ENABLED=false
23+
24+
CMD python main.py

‎scenarios/python_live_heap_3.15/README.md‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
## Live-Heap Profiling (3.15 candidate)
2+
3+
Validates that the Datadog Python profiler's **persistent live-heap profile**
4+
correctly reports the set of live (still-allocated) sampled objects. This is the
5+
**3.15 candidate** half of the `3.14 -> 3.15` migration pair (see
6+
`python_live_heap_3.14`).
7+
8+
Unlike `alloc-space`/`alloc-samples` (which count every allocation over the
9+
interval), the live-heap profile is a running snapshot of what is *currently
10+
live*: allocations are added when sampled and subtracted when freed, and the
11+
snapshot is exported non-destructively on every upload. It is attached as a
12+
second pprof (`<prefix>.<pid>.<seq>.heap.pprof`) alongside the primary profile.
13+
14+
## Test Application
15+
16+
`main.py` retains a known live set for the whole run, split into two
17+
distinctly-named call sites that allocate **equal-size** objects (16 KiB) and
18+
differ only in count:
19+
20+
- `retain_major` - 1,600 objects (~25 MiB, ~80% of the live set)
21+
- `retain_minor` - 400 objects (~6 MiB, ~20% of the live set)
22+
23+
Equal sizes mean the 80/20 split holds for both metrics the live-heap profile
24+
reports: `heap-space` (live bytes = count x size) and `heap-live-samples` (live
25+
object count). It uses `bytes` (`PyObject_Malloc`, the OBJ allocator domain) so
26+
the heap profiler tracks the objects identically across versions, independent of
27+
the `DD_PROFILING_MEMORY_MEM_DOMAIN_ENABLED` toggle (`bytearray` moved OBJ -> MEM
28+
in 3.13). The objects are held alive while the process idles through several
29+
upload intervals (`DD_PROFILING_UPLOAD_INTERVAL=5`), so each exported heap
30+
snapshot contains the full live set.
31+
32+
## Expected Profile
33+
34+
Assertions run against the heap snapshot pprof only (selected with
35+
`pprof-regex`). `DD_PROFILING_HEAP_SAMPLE_SIZE=16384` (== object size) yields
36+
roughly one sample per object, keeping the per-stack proportions stable.
37+
38+
- `heap-space` (live bytes):
39+
- `^<module>;Target.run;Target.retain_major$` ~= 80%
40+
- `^<module>;Target.run;Target.retain_minor$` ~= 20%
41+
- `heap-live-samples` (live object count): same ~80/20 split.
42+
43+
`allow_first_profile_failure` tolerates the first snapshot, which may be taken
44+
before the live set is fully built.
45+
46+
## Notes
47+
48+
Requires a **dd-trace-py wheel** at image build time (set `DDTRACE_INSTALL_URL`,
49+
as `downstream-python.yml` does). Do **not** `pip install ddtrace` from PyPI —
50+
3.15 wheels may not be published yet. Excluded from prof-correctness `main` CI
51+
until 3.15 wheels are generally available; runs via the dd-trace-py downstream
52+
gate.

‎scenarios/python_live_heap_3.15/expected_profile.json‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
{
2+
"test_name": "python_live_heap",
3+
"note": "Persistent live-heap profile. Assertions run against the separate heap snapshot pprof (<prefix>.<pid>.<seq>.heap.pprof), selected via pprof-regex. heap-space = live bytes, heap-live-samples = live object count. retain_major/retain_minor allocate equal-size objects in an 80/20 count split, so both metrics show the same ~80/20 proportions. Thread-name labels are intentionally not asserted: the heap profiler tags samples with the numeric OS thread id, which varies per run.",
4+
"pprof-regex": ".*heap\\.pprof",
5+
"scale_by_duration": false,
6+
"allow_first_profile_failure": true,
7+
"stacks": [
8+
{
9+
"profile-type": "heap-space",
10+
"stack-content": [
11+
{
12+
"regular_expression": "^<module>;Target\\.run;Target\\.retain_major$",
13+
"percent": 80,
14+
"error_margin": 10,
15+
"labels": []
16+
},
17+
{
18+
"regular_expression": "^<module>;Target\\.run;Target\\.retain_minor$",
19+
"percent": 20,
20+
"error_margin": 10,
21+
"labels": []
22+
}
23+
]
24+
},
25+
{
26+
"profile-type": "heap-live-samples",
27+
"stack-content": [
28+
{
29+
"regular_expression": "^<module>;Target\\.run;Target\\.retain_major$",
30+
"percent": 80,
31+
"error_margin": 12,
32+
"labels": []
33+
},
34+
{
35+
"regular_expression": "^<module>;Target\\.run;Target\\.retain_minor$",
36+
"percent": 20,
37+
"error_margin": 12,
38+
"labels": []
39+
}
40+
]
41+
}
42+
]
43+
}

0 commit comments

Comments
 (0)