Skip to content

[DEMO] Normal PR control (perf-gate GREEN)#6

Open
Neil4561 wants to merge 6 commits into
unified-POCfrom
neil/perf-demo-green-noop
Open

[DEMO] Normal PR control (perf-gate GREEN)#6
Neil4561 wants to merge 6 commits into
unified-POCfrom
neil/perf-demo-green-noop

Conversation

@Neil4561

@Neil4561 Neil4561 commented Jun 25, 2026

Copy link
Copy Markdown

Demo PR (not for merge)

GREEN case for the performance smoke-test demo.

Change: documentation only. This PR expands the benchmark_non_rl.py module docstring and does not change benchmark behavior or runtime code.

Why this case exists: this is the healthy control. It shows what a normal PR should look like when the smoke test runs: most cells should stay near or above their historical baselines.

Matrix: Isaac-Cartpole-Direct + Isaac-Factory-GearMesh-Direct-v0 (physx) + Isaac-Velocity-Flat-G1-v0.

Trigger via mirror to pull-request/<N> or workflow_dispatch.

Neil4561 added 2 commits June 25, 2026 18:17
Trim the benchmark matrix to the two well-baselined demo tasks (physx +
newton each) for a fast, clean four-cell verdict table. Demo only; not for
merge.
Expand the one-line module docstring to describe what the benchmark
measures (per-step frametimes -> effective environment FPS). Documentation
only; no behavior change. Serves as the GREEN/normal-PR control in the perf
smoke-test demo.
Include the newly-seeded Factory-GearMesh task so the verdict table shows a
third passing task alongside Cartpole and G1. Demo only; not for merge.
@github-actions

github-actions Bot commented Jun 25, 2026

Copy link
Copy Markdown

Performance Smoke Results

Baseline read SHA: fbacaae09d4e

Run context

  • GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition
  • Runtime: cuda=12.8, driver=595.71.05, warp=1.13.0
  • Threshold policy: rolling baseline per row. Each task/backend is compared against its own compatible baseline samples using median/MAD.

How to read this table

  • PASS: no meaningful slowdown was detected.
  • WARN: suspicious or uncertain result, such as insufficient baselines, retry-only success, or a slowdown in the WARN band.
  • BLOCK: a blocking-level regression signal was detected. In this POC, BLOCK is advisory (it flags a regression) and only fails the check when the gate is explicitly configured as blocking.
  • HARD_FAILURE: the benchmark did not produce usable FPS data, for example an import/init/runtime failure or config mismatch.
  • Delta: change in FPS versus the baseline. + means faster than baseline (speedup); - means slower than baseline (slowdown).
  • Threshold policy: explains which comparison rule produced the verdict. rolling_window means each row uses its own compatible baseline median/MAD window, not one shared FPS cutoff.
  • Noise: baseline MAD as a percent of baseline FPS. Higher noise means the task/backend naturally varies more from run to run.
  • Phase: the stage a HARD_FAILURE happened in (e.g. import, init, runtime); blank when the benchmark ran to completion.
  • Retried: yes if the cell only passed after an automatic re-run.
Task Backend Verdict FPS Baseline Delta (+ faster / - slower) Noise Samples Phase Notes Retried
Isaac-Cartpole-Direct newton PASS 1013144.9 1013835.8 -0.07% 1.11% 6 p99/med=1.178; outliers=0 no
Isaac-Cartpole-Direct physx PASS 278663.0 281551.3 -1.03% 0.97% 6 p99/med=1.316; outliers=0 no
Isaac-Factory-GearMesh-Direct-v0 physx PASS 1228.9 1209.0 +1.65% 0.21% 6 p99/med=1.031; outliers=1 no
Isaac-Velocity-Flat-G1-v0 newton BLOCK 30173.5 33462.9 -9.83% 0.67% 6 block_confirmed(n=1); p99/med=1.133; outliers=0 no
Isaac-Velocity-Flat-G1-v0 physx PASS 10613.3 10770.8 -1.46% 1.39% 6 p99/med=1.167; outliers=0 no

Updated for 74bab6f · run

Neil4561 added 3 commits June 26, 2026 08:19
Add a demo-branch confirmation pass that reruns only cells that
initially block and evaluates the median attempt before posting the
final sticky summary. This keeps the clean demo PR from relying on a
single noisy benchmark draw while preserving the old behavior if no
block appears.
Resolve confirmation attempt artifact directories before mounting them
into Docker so reruns can write benchmark outputs successfully.
Bump the demo task matrix to baseline epoch 2 so the clean PR can
compare against freshly seeded samples instead of the noisy earlier
baseline window.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant