chore(ci): Prototype: SMP client-side report rendering by Arpafaucon · Pull Request #2324 · DataDog/saluki

Arpafaucon · 2026-08-13T08:09:49Z

Summary

Use a SMP binary generated by https://github.com/DataDog/single-machine-performance/pull/4693/s

Change Type

Bug fix
New feature
Non-functional (chore, refactoring, docs)
Performance

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8948e88e8b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-08-13T08:31:05Z

    when: always
  script:
-    - python3 ./ci/tooling/build-smp-report.py
+    - ./smp report render


Make the SMP binary available to the report job

On every development-branch pipeline, report-benchmarks-adp starts in a fresh container and invokes ./smp, but the SMP CI image does not install that binary and the upstream job's artifact list contains only submission_metadata and the two report files. The binary downloaded by run-benchmarks-adp therefore does not reach this job, so this command exits with “No such file or directory” before generating or posting the condensed report, causing the reporting job—and otherwise successful pipelines—to fail. Download smp here or publish it as an artifact needed by this job.

Useful? React with 👍 / 👎.

datadog-datadog-prod-us1

The standalone report job cannot invoke the new renderer because its fresh workspace receives only report artifacts and the CI image does not include smp; every PR benchmark report therefore stops before posting its condensed comment.

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit 8948e88 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

datadog-datadog-prod-us1 · 2026-08-13T08:32:46Z

    when: always
  script:
-    - python3 ./ci/tooling/build-smp-report.py
+    - ./smp report render


Acquire smp in the standalone report job

Every development-branch benchmark report job fails before rendering, so PR authors receive no condensed benchmark comment even after a successful benchmark sync.

Assertion details

Input: Any development-branch pipeline that schedules the separate report-benchmarks-adp job.

Expected: The reporting job should acquire the pinned SMP binary before invoking smp report render, then publish the condensed PR comment.

Actual: report-benchmarks-adp invokes ./smp, but neither the job image nor the artifacts downloaded through needs contain that executable.

Suggested change

- ./smp report render

- *setup-smp-env

- ./ci/tooling/configure-smp-aws-credentials.sh

- aws --profile ${AWS_NAMED_PROFILE} s3 cp s3://smp-cli-releases/${SMP_VERSION}/x86_64-unknown-linux-musl/smp smp && chmod +x smp

- ./smp report render

Was this helpful? React 👍 or 👎
_{🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

pr-commenter · 2026-08-13T09:16:47Z

Binary Size Analysis (Agent Data Plane)

Baseline: ca31d5f · Comparison: d570b7f · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 41.48 MiB (baseline) vs 41.48 MiB (comparison)
Size Change: +8 B (+0.00%)

✅ Binary size difference within threshold

Changes by Module

Module	File Size	Symbols
`anon.dca7d28c8f462601bb81d85101fb8da4.1.llvm.14939170826646974924`	+130 B	1
`anon.dca7d28c8f462601bb81d85101fb8da4.1.llvm.7079599386923508682`	-129 B	1
`anon.dca7d28c8f462601bb81d85101fb8da4.4.llvm.14939170826646974924`	+114 B	1
`anon.dca7d28c8f462601bb81d85101fb8da4.4.llvm.7079599386923508682`	-113 B	1
`anon.dca7d28c8f462601bb81d85101fb8da4.3.llvm.14939170826646974924`	+109 B	1
`anon.dca7d28c8f462601bb81d85101fb8da4.3.llvm.7079599386923508682`	-108 B	1
`anon.dca7d28c8f462601bb81d85101fb8da4.0.llvm.14939170826646974924`	+97 B	1
`anon.dca7d28c8f462601bb81d85101fb8da4.0.llvm.7079599386923508682`	-96 B	1
`anon.dca7d28c8f462601bb81d85101fb8da4.2.llvm.14939170826646974924`	+95 B	1
`anon.dca7d28c8f462601bb81d85101fb8da4.2.llvm.7079599386923508682`	-94 B	1
`[Unmapped]`	+3 B	1

Detailed Symbol Changes

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  [NEW]    +130  [NEW]     +40    anon.dca7d28c8f462601bb81d85101fb8da4.1.llvm.14939170826646974924
  [NEW]    +114  [NEW]     +24    anon.dca7d28c8f462601bb81d85101fb8da4.4.llvm.14939170826646974924
  [NEW]    +109  [NEW]     +19    anon.dca7d28c8f462601bb81d85101fb8da4.3.llvm.14939170826646974924
  [NEW]     +97  [NEW]      +7    anon.dca7d28c8f462601bb81d85101fb8da4.0.llvm.14939170826646974924
  [NEW]     +95  [NEW]      +5    anon.dca7d28c8f462601bb81d85101fb8da4.2.llvm.14939170826646974924
  +6.8%      +3  [ = ]       0    [Unmapped]
  [DEL]     -94  [DEL]      -5    anon.dca7d28c8f462601bb81d85101fb8da4.2.llvm.7079599386923508682
  [DEL]     -96  [DEL]      -7    anon.dca7d28c8f462601bb81d85101fb8da4.0.llvm.7079599386923508682
  [DEL]    -108  [DEL]     -19    anon.dca7d28c8f462601bb81d85101fb8da4.3.llvm.7079599386923508682
  [DEL]    -113  [DEL]     -24    anon.dca7d28c8f462601bb81d85101fb8da4.4.llvm.7079599386923508682
  [DEL]    -129  [DEL]     -40    anon.dca7d28c8f462601bb81d85101fb8da4.1.llvm.7079599386923508682
  +0.0%      +8  [ = ]       0    TOTAL

pr-commenter · 2026-08-13T14:15:17Z

Regression Detector (Agent Data Plane)

this test report is locally rendered
Run ID: 5423a86a-c49e-4aed-b976-bef1c294ca93
Baseline: ca31d5fc · Comparison: d570b7fa · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (5)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment	goal	Δ mean %	links
quality_gates_rss_dsd_medium	memory	⚪ +0.30	(metrics) (profiles) (logs)
quality_gates_rss_dsd_heavy	memory	⚪ +0.22	(metrics) (profiles) (logs)
quality_gates_rss_dsd_low	memory	⚪ +0.05	(metrics) (profiles) (logs)
quality_gates_rss_idle	memory	⚪ -0.11	(metrics) (profiles) (logs)
quality_gates_rss_dsd_ultraheavy	memory	⚪ -0.45	(metrics) (profiles) (logs)

Bounds Checks: ✅ Passed (5)

experiment	check	replicates	observed	links
quality_gates_rss_dsd_heavy	memory_usage	20/20	✅ 240005120 ≤ 262144000	(metrics) (profiles) (logs)
quality_gates_rss_dsd_low	memory_usage	20/20	✅ 53981184 ≤ 62914560	(metrics) (profiles) (logs)
quality_gates_rss_dsd_medium	memory_usage	20/20	✅ 94736384 ≤ 104857600	(metrics) (profiles) (logs)
quality_gates_rss_dsd_ultraheavy	memory_usage	20/20	✅ 405168128 ≤ 440401920	(metrics) (profiles) (logs)
quality_gates_rss_idle	memory_usage	20/20	✅ 33251328 ≤ 41943040	(metrics) (profiles) (logs)

Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.

dd-octo-sts Bot added the area/ci CI/CD, automated testing, etc. label Aug 13, 2026

Arpafaucon changed the title ~~[SMP] Prototype: client-side report rendering~~ chore(SMP) Prototype: client-side report rendering Aug 13, 2026

Arpafaucon force-pushed the greg/smp-ci-reporting branch from c4803f8 to 8948e88 Compare August 13, 2026 08:15

Arpafaucon changed the title ~~chore(SMP) Prototype: client-side report rendering~~ chore(SMP): Prototype: client-side report rendering Aug 13, 2026

Arpafaucon changed the title ~~chore(SMP): Prototype: client-side report rendering~~ chore(ci): Prototype: SMP client-side report rendering Aug 13, 2026

Arpafaucon marked this pull request as ready for review August 13, 2026 08:29

Arpafaucon requested a review from a team as a code owner August 13, 2026 08:29

chatgpt-codex-connector Bot reviewed Aug 13, 2026

View reviewed changes

datadog-datadog-prod-us1 Bot reviewed Aug 13, 2026

View reviewed changes

Arpafaucon marked this pull request as draft August 13, 2026 09:08

prototype - use smp report render to generate PR comment

45c414a

Arpafaucon force-pushed the greg/smp-ci-reporting branch from 8948e88 to 45c414a Compare August 13, 2026 09:36

Arpafaucon added 4 commits August 13, 2026 11:39

download SMP binary

3361dd9

fix silly mkdir mistake

fab8295

fix AWS reqs

c9d44ba

fix paths ?

dee94fd

Arpafaucon added 2 commits August 13, 2026 16:38

fix report

e3ba4f9

fix template - remove duplicate title

d570b7f

Navigation Menu

Uh oh!

chore(ci): Prototype: SMP client-side report rendering - #2324

chore(ci): Prototype: SMP client-side report rendering#2324
Arpafaucon wants to merge 7 commits into
mainfrom
greg/smp-ci-reporting

Arpafaucon commented Aug 13, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Aug 13, 2026

Uh oh!

datadog-datadog-prod-us1 Bot left a comment

Uh oh!

datadog-datadog-prod-us1 Bot Aug 13, 2026

Uh oh!

pr-commenter Bot commented Aug 13, 2026 •

edited

Loading

Uh oh!

pr-commenter Bot commented Aug 13, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

-    - ./smp report render
+    - *setup-smp-env
+    - ./ci/tooling/configure-smp-aws-credentials.sh
+    - aws --profile ${AWS_NAMED_PROFILE} s3 cp s3://smp-cli-releases/${SMP_VERSION}/x86_64-unknown-linux-musl/smp smp && chmod +x smp
+    - ./smp report render

Uh oh!

Conversation

Arpafaucon commented Aug 13, 2026

Summary

Change Type

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

datadog-datadog-prod-us1 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

datadog-datadog-prod-us1 Bot Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

pr-commenter Bot commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Binary Size Analysis (Agent Data Plane)

✅ Binary size difference within threshold

Uh oh!

pr-commenter Bot commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Regression Detector (Agent Data Plane)

Optimization Goals: ✅ No significant changes detected

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

pr-commenter Bot commented Aug 13, 2026 •

edited

Loading

pr-commenter Bot commented Aug 13, 2026 •

edited

Loading