test(profiling): unflake `test_copy_memory_stats` by KowalskiThomas · Pull Request #19706 · DataDog/dd-trace-py

KowalskiThomas · 2026-08-14T14:49:39Z

Description

This PR attempts to unflake test_copy_memory_stats which has been rarely (but still) failing lately. I suspect this is because there is a race between the sampling thread upgrading to fast copy memory and the uploader thread dumping stats to the file (which would explain why it's so rare).

Something LLMs also suggested is to make sure we sort file names by their actual order and not just with sorted, since sorted would sort lexicographically in which case profile.10 would appear before profile.2, so I also fixed that.

Fixes DD_QMLVTC DD_KFTYAJ

datadog-prod-us1-3 · 2026-08-14T14:49:46Z

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 09e9ae0 | Docs | Datadog PR Page | Give us feedback!}

cit-pr-commenter-54b7da · 2026-08-14T15:00:05Z

Codeowners resolved as

Resolved from the full PR diff against main using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

No remaining files require a CODEOWNERS review.

cit-pr-commenter-54b7da · 2026-08-14T15:00:24Z

Dependency direction analysis

⚠️ Existing dependency direction violations

There are 254 dependency direction violations that already exist on the base branch and have not been changed by this PR.

Show existing violations (showing 5 of 254 highest severity)

ddtrace.internal.tracemethods -×-> ddtrace.trace  (internal-core -> product:tracing, score=134)
ddtrace.llmobs._integrations.langgraph -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=132)
ddtrace.llmobs._integrations.langchain -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=132)
ddtrace.debugging._exception.replay -×-> ddtrace.trace  (product:debugging -> product:tracing, score=132)
ddtrace.llmobs._integrations.base -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=132)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:
uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json

cit-pr-commenter-54b7da · 2026-08-14T15:00:32Z

Circular import analysis

⚠️ Existing circular imports

There are 5 circular imports that already exist on the base branch and have not been changed by this PR.

ddtrace.contrib.internal.pytorch._distributed -> ddtrace.contrib.internal.pytorch._rank_root -> ddtrace.contrib.internal.pytorch._distributed
ddtrace.contrib.internal.django.patch -> ddtrace.contrib.internal.django.response -> ddtrace.contrib.internal.django.patch
ddtrace.errortracking._handled_exceptions.bytecode_injector -> ddtrace.errortracking._handled_exceptions.callbacks -> ddtrace.errortracking._handled_exceptions.collector -> ddtrace.errortracking._handled_exceptions.bytecode_reporting -> ddtrace.errortracking._handled_exceptions.bytecode_injector
ddtrace.llmobs -> ddtrace.llmobs._evaluators -> ddtrace.llmobs._evaluators.format -> ddtrace.llmobs._experiment -> ddtrace.llmobs
ddtrace.appsec._asm_request_context -> ddtrace.appsec._iast._iast_request_context_base -> ddtrace.appsec._iast._iast_env -> ddtrace.appsec._iast.reporter -> ddtrace.appsec._exploit_prevention.stack_traces -> ddtrace.appsec._asm_request_context

taegyunkim

Yeah I remember that we had to properly sort the pprof files as well.

KowalskiThomas · 2026-08-18T06:31:36Z

/merge -f --reason "test only"

gh-worker-devflow-routing-ef8351 · 2026-08-18T06:31:41Z

View all feedbacks in Devflow UI.

2026-08-18 06:31:40 UTC ℹ️ Start processing command /merge -f --reason "test only"

2026-08-18 06:31:47 UTC ℹ️ MergeQueue: Pull request is not mergeable yet

It will be processed automatically as soon as GitHub reports it as mergeable. View in MergeQueue UI.

Run /code blockers to see what is blocking it.
Run /remove to cancel it.

2026-08-18 06:43:31 UTC ℹ️ MergeQueue: merge request added to the queue without running the checks

This merge request will be merged without running the pre merge checks for of the following reason: test only.

2026-08-18 06:43:39 UTC ℹ️ MergeQueue: This merge request was merged

Warning

This change was merged without running any pre merge CI checks

Reason: test only

KowalskiThomas · 2026-08-18T06:32:51Z

/gitlab resync-job-status

gh-worker-devflow-routing-ef8351 · 2026-08-18T06:37:13Z

View all feedbacks in Devflow UI.

2026-08-18 06:32:54 UTC ℹ️ Start processing command /gitlab resync-job-status
If you need support, contact us on Slack #ci-infra-support!

2026-08-18 06:37:11 UTC 🚨 Devflow

404 Not Found

Details

child workflow execution error (type: cirel.Cirel_SyncCommitJobsStatusWorkflow, workflowID: d2869553-2cba-469c-9080-f0fa1ed87a1b_44, runID: 01a01392-af7b-7135-a710-412c8604be7d, initiatedEventID: 44, startedEventID: 45): activity error (type: gitlab.GitlabService_GetJobsByCommit, scheduledEventID: 14, startedEventID: 15, identity: 1@gitlab-worker-569fbc4db8-t7csm@): 404 Not Found

If you need support, contact us on Slack #ci-infra-support with those details!

## Description This PR attempts to unflake `test_copy_memory_stats` which has been rarely (but still) failing lately. I suspect this is because there is a race between the sampling thread upgrading to fast copy memory and the uploader thread dumping stats to the file (which would explain why it's so rare). Something LLMs also suggested is to make sure we sort file names by their actual order and not just with `sorted`, since `sorted` would sort lexicographically in which case `profile.10` would appear before `profile.2`, so I also fixed that. Fixes DD_QMLVTC DD_KFTYAJ Co-authored-by: thomas.kowalski <thomas.kowalski@datadoghq.com>

test(profiling): unflake test_copy_memory_stats

2147072

KowalskiThomas added changelog/no-changelog A changelog entry is not required for this PR. Profiling Continous Profling labels Aug 14, 2026

KowalskiThomas marked this pull request as ready for review August 14, 2026 15:02

KowalskiThomas requested a review from a team as a code owner August 14, 2026 15:02

KowalskiThomas requested a review from vlad-scherbich August 14, 2026 15:02

taegyunkim approved these changes Aug 14, 2026

View reviewed changes

vlad-scherbich approved these changes Aug 17, 2026

View reviewed changes

Comment thread tests/profiling/collector/pprof_utils.py

Comment thread tests/profiling/collector/test_copy_memory_stats.py

KowalskiThomas force-pushed the kowalski/test-profiling-unflake-test_copy_memory_stats branch 2 times, most recently from ca666a7 to 28f32d6 Compare August 17, 2026 12:41

test: use correct file name sorting

09e9ae0

KowalskiThomas force-pushed the kowalski/test-profiling-unflake-test_copy_memory_stats branch from 28f32d6 to 09e9ae0 Compare August 17, 2026 13:21

gh-worker-dd-mergequeue-cf854d Bot merged commit b1b43b2 into main Aug 18, 2026
130 of 164 checks passed

gh-worker-dd-mergequeue-cf854d Bot deleted the kowalski/test-profiling-unflake-test_copy_memory_stats branch August 18, 2026 06:43

Navigation Menu

Uh oh!

test(profiling): unflake `test_copy_memory_stats` - #19706

test(profiling): unflake `test_copy_memory_stats`#19706
gh-worker-dd-mergequeue-cf854d[bot] merged 2 commits into
mainfrom
kowalski/test-profiling-unflake-test_copy_memory_stats

KowalskiThomas commented Aug 14, 2026 •

edited

Loading

Uh oh!

datadog-prod-us1-3 Bot commented Aug 14, 2026 •

edited

Loading

Uh oh!

cit-pr-commenter-54b7da Bot commented Aug 14, 2026 •

edited

Loading

Uh oh!

cit-pr-commenter-54b7da Bot commented Aug 14, 2026 •

edited

Loading

Uh oh!

cit-pr-commenter-54b7da Bot commented Aug 14, 2026 •

edited

Loading

Uh oh!

taegyunkim left a comment

Uh oh!

Uh oh!

Uh oh!

KowalskiThomas commented Aug 18, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Aug 18, 2026 •

edited

Loading

Uh oh!

KowalskiThomas commented Aug 18, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Aug 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

KowalskiThomas commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

datadog-prod-us1-3 Bot commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cit-pr-commenter-54b7da Bot commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codeowners resolved as

Uh oh!

cit-pr-commenter-54b7da Bot commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dependency direction analysis

⚠️ Existing dependency direction violations

Uh oh!

cit-pr-commenter-54b7da Bot commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Circular import analysis

⚠️ Existing circular imports

Uh oh!

taegyunkim left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

KowalskiThomas commented Aug 18, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KowalskiThomas commented Aug 18, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Aug 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

KowalskiThomas commented Aug 14, 2026 •

edited

Loading

datadog-prod-us1-3 Bot commented Aug 14, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Aug 14, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Aug 14, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Aug 14, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Aug 18, 2026 •

edited

Loading