Add perf-smoke comparator scaffolding under tools/#1
Draft
Neil4561 wants to merge 3 commits into
Draft
Conversation
github-actions
Bot
added
documentation
Add a PR-time performance gate that benchmarks a small fixed set of Isaac Lab tasks on the arm64 L40S fleet and compares each result against a rolling baseline, reporting PASS/WARN/BLOCK back onto the PR. Includes the gate orchestrator, the runtime-free comparator (median+MAD) with its unit and stress tests, the per-task baseline config and seeded rolling-window history, re-baselining helpers, the Warp/replicator arm64 shim, and the GitHub Actions workflow (push to pull-request/* with copy-pr-bot, per-task matrix, commit-status reporting). Advisory to start.
Reconcile the perf smoke gate with the current develop registrations and benchmark provenance, surfaced by a 1-task L40S dry run: - Point the cartpole baseline keys at the registered Isaac-Cartpole gym id via task_id (the manager-based env dropped its -v0 suffix), keeping the stable history/baseline keys intact. - Record the physics= backend in benchmark_info (new get_physics_string) so the comparator can verify the simulation backend. Previously only presets= was echoed, so a physics-only task reported presets=default and the config check spuriously hard-failed. - Split the comparator config check: physics= is matched against benchmark_info.physics, presets= against benchmark_info.presets. - Re-baseline Isaac-Cartpole (physx) to the current build (~276k FPS on L40S); the prior 115k was ~2.4x stale. Marked provisional pending a full re-baseline on the runner fleet. - Skip the get-pr-info step on workflow_dispatch (no originating PR), so manual single-task trial runs are not blocked.
Finish env-fingerprint bucketing and per-sample provenance for the rolling-window history, add unit tests for the orchestrator, rebaseline tool, and fingerprint/provenance helpers, and document the architecture in DESIGN.md.
Neil4561
force-pushed
the
neilm/perf-smoke-gate
branch
from
June 15, 2026 22:53
633e5ec to
1d977c3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Phase 1 groundwork for the PR-time performance regression smoke gate.
Adds a runtime-free comparator (
tools/perf_smoke/check_perf_regression.py) that reads an OmniPerf benchmark JSON, looks up the per-task per-GPU baseline, and exits with one of three codes that the eventual CI workflow will translate into a PR signal:0PASS1REGRESSIONmax_regression_pct.2HARD_FAILUREREGRESSIONso environment-breakage incidents (e.g. thetomllibimport failure on the wrong Python) surface as a different signal than a real perf regression.The comparator has no IsaacLab / torch / Kit dependencies, runs from a stock Python ≥ 3.10, and writes a structured single-line summary to stdout plus an optional
$GITHUB_STEP_SUMMARYblock.Phase 1 baseline:
Isaac-Cartpole-Direct-v0onNVIDIA L40(num_envs=4096,num_frames=100, default preset). The baseline FPS is the P25 of the OmniPerf historical WARM distribution over a 77-day stable window (n=509, pooled CV 2.62%). The 10% tolerance is intentionally loose for the warning-only Phase 1 and absorbs the sporadic 12–17% contention drops observed roughly 1-in-30 historical runs.Tests use stdlib
unittest(notpytest) becausetools/conftest.pyblocks pytest collection beneathtools/. 25 cases cover pass / regression / improvement / threshold boundary plus 15+ structural failure modes. Seetools/perf_smoke/README.mdfor the full contract.CI integration is intentionally deferred — its shape depends on the runner-platform decision (GH self-hosted vs OSMO vs Colossus). This PR deliberately builds only the platform-independent piece so the workflow wiring can be added in a follow-up without churn.
Draft because the workflow wiring lands in a separate commit on this branch after the runner-platform decision.
Type of change
Test plan
python tools/perf_smoke/test_check_perf_regression.py(Windows, Python 3.14) — 25/25 passedpython3 tools/perf_smoke/test_check_perf_regression.py(Linux, Python 3.10) — 25/25 passed./isaaclab.sh -p tools/perf_smoke/test_check_perf_regression.py(Linux, IsaacLab venv on Python 3.11) — 25/25 passed./isaaclab.sh -f(Linux) — all pre-commit hooks passed, no files modifiedScreenshots
N/A — pure tooling change, no UI.
Checklist
pre-commitchecks with./isaaclab.sh --formattools/perf_smoke/README.md)unittestcases)config/extension.tomlfile — N/A:tools/is not a versioned package, so no changelog fragment applies (perAGENTS.md)CONTRIBUTORS.mdor my name already exists there