Rewrite benchmark user guide by AntoineRichard · Pull Request #6595 · isaac-sim/IsaacLab

AntoineRichard · 2026-07-17T21:25:45Z

Description

Rewrites the benchmark documentation around the public CLI workflows and the
questions benchmark users need to answer:

choose between runtime, startup, training, and play measurements;
emit both readable terminal summaries and machine-readable schema output;
use warm-up steps and workload sizes that produce representative results;
interpret collection, environment-step, total, startup, rendering, and
resource metrics;
reproduce published measurements on the standard workstation while recording
CPU, GPU, software, task, seed, and revision provenance.

The guide uses Cartpole throughout and includes collapsed workstation output for
each workflow. The runtime example also shows rendered camera timing. It explains
that Isaac Lab overhead measurement forces synchronization, can materially slow
Newton, and must not be enabled for throughput comparisons.

The documentation is split into a CLI-first user guide and an advanced benchmark
framework/API reference. The micro-benchmark guide is reorganized alongside it
to explain why isolated benchmarks exist, how to run them, and how to interpret
their output.

This branch also fixes two provenance fields used by the examples:

RSL-RL training schemas now record the generated checkpoint path.
Startup schemas now record the configured environment count.

Existing smoke tests were strengthened to check both fields; no new test suite
was added.

Review boundary and dependencies

This is a stacked draft. The documentation commits begin at 2493f14f094 and
end at 64a9295eb26. The schema provenance fix is f02ca894a91.

It depends on the recent benchmark implementation and micro-benchmark PR stack:

The expanded aggregate diff against develop includes those dependencies; they
are not part of the documentation review boundary above.

Workstation captures

The embedded examples were captured at f02ca894a91 on the standard NVIDIA
workstation using Isaac Sim 6.0.0.1 and Torch 2.11.0+cu128. The guide documents
the repository SSH helper and requires published benchmark results to come from
that workstation. It also calls out CPU frequency policy and competing workloads
because both can materially affect the results.

Type of change

Bug fix (non-breaking change which fixes an issue)
Documentation update

Screenshots

Not applicable. The output examples are rendered as collapsed documentation
sections.

Validation

./isaaclab.sh -p -m pytest scripts/benchmarks/test/test_startup_smoke.py scripts/benchmarks/test/test_benchmark_smoke.py -k "startup_writes_startup_bundle or (training_and_play_write_bundles and rsl_rl)"
- 2 passed, 19 deselected
./isaaclab.sh -f
./isaaclab.sh -d
Fresh local runtime probe in a clean uv environment
Runtime, startup, training, and trained-policy play captures on the workstation

Checklist

I have read and understood the contribution guidelines
I have run the pre-commit checks with ./isaaclab.sh -f
I have made corresponding changes to the documentation
My changes generate no new documentation warnings
I have updated existing tests that prove the provenance fixes are effective
I have updated the existing Isaac Lab changelog fragment
My name already exists in CONTRIBUTORS.md

Exclude configurable warmup frames from runtime metrics and compute throughput from total measured work over total wall time. Use longer defaults so the quick benchmark is less sensitive to startup and short sampling windows.

Add an opt-in runtime benchmark probe around the environment's SimulationContext step calls. Report host time inside and outside that boundary while preserving the original method after measurement.

Measure environment and synchronized simulation stepping across runtime, training, and play benchmarks. Store the resulting Isaac Lab overhead in benchmark schema 1.1 while leaving regular training and play entry points uninstrumented.

Require an explicit benchmark flag before wrapping environment and simulation steps. Disabled benchmarks now avoid method replacement and device synchronization while schema 1.1 continues to represent measured runs.

Delegate synchronization and elapsed-time measurement to the existing Isaac Lab Timer. Keep the interception layer focused on recording per-step samples so environment, simulation, and derived overhead metrics include standard deviation and peak values.

Record the env.step boundary in every benchmark workflow without adding explicit synchronization. Keep the nested synchronized simulation probe optional, and expose environment-step FPS in schema 1.2 so runtime, play, and training results can be compared on the same scope.

Name the opt-in instrumentation after the Isaac Lab overhead it reports instead of the implementation detail used to derive it.

Synchronize before environment and simulation timing begins so policy and action-generation work queued earlier is not attributed to Isaac Lab overhead.

Recalculate throughput standard deviation around the aggregate effective mean so benchmark MeanStd values use one consistent center.

Validate workload counts before launch and require positive environment timing samples so empty synchronized runs fail with actionable errors instead of division by zero.

Run the training and play smoke matrix without synchronized overhead measurement, retain one focused synchronized replay, and verify pending work before env.step is excluded from its metric.

Keep the first runtime step outside throughput regardless of overhead instrumentation and floor derived overhead at zero when nested wall timers cross due to measurement noise.

Rename the disputed overhead metric as a serialized synchronized step breakdown and expose explicit measurement modes. Mark every instrumented timing and rate as diagnostic, validate the additive partition, and document why outside-simulation time is not removable overhead.

Follow-up polish from review of the runtime/synchronized-timing changes; no behavior change. - Drop the redundant ``dest=``/``default=False`` on runtime.py's ``--measure_synchronized_step_breakdown`` (argparse derives both) so the flag matches the eight RL adapter scripts. - Remove the single-use ``timer_context`` alias and enter the step-timing recorder alongside ``BenchmarkMonitor`` on one line, matching the adapters' context-manager pattern. - Reorder the ``build_runtime`` Args block to signature order (``aggregate_throughput`` before ``frames_per_environment_step``). - Clarify ``MeanStd.mean``/``std``: for effective-throughput fields these are the aggregate rate and the dispersion about the effective mean, not the arithmetic sample mean and standard deviation.

Bring PR isaac-sim#6553 on top of the benchmark timing work from PR isaac-sim#6474 so the benchmark library can dispatch the new RL entrypoints directly.

Move the supported benchmark workflows into isaaclab.benchmark and expose typed in-process dispatch for runtime, startup, training, and play benchmarks. Keep deprecated script and test namespace shims so existing callers can migrate without an immediate break.

Use one device-wide synchronization per timing boundary and keep mock backend state preallocated between samples. Initialize ordering and dynamics buffers so PhysX and Newton accessors are measured without hidden skips or allocation work.

Replace per-file Python shims with lazy module proxies so legacy imports remain compatible while Git can recognize the benchmark implementation moves. Retain type forwarding stubs and document removal in Isaac Lab 3.1.

Add a warmup_steps parameter to EnvironmentStepTimingRecorder that drops the first N ``env.step()`` calls (CUDA graph capture, lazy kernel compilation) from the recorded environment-step timing. In serialized mode the skipped steps' simulation call-count and time are rolled back so the recorded aggregate stays consistent. Expose it as an opt-in ``--warmup_steps`` flag (default 0, validated non-negative, recorded in metadata) on the eight RL train/play benchmark adapters. ``runtime.py`` is unchanged: its ``--warmup_frames`` already excludes the startup step before the recorder is installed. Add recorder unit tests for the host-return and serialized exclusion paths and a parse-level test rejecting a negative ``--warmup_steps``.

Preserve the destination Warp dtype when copying mock binding data so structured pose and spatial-vector buffers can be exercised by asset data benchmarks.

Cover articulation, rigid object, and rigid object collection method and data access performance using kitless mock bindings comparable to the Newton benchmark matrix.

Apply the repository Ruff format to the newly tracked benchmark scripts and keep fill-ratio selection conditions within the line-length limit.

Measure synchronized updates, submission latency, and backend reads for contact, frame-transformer, IMU, PVA, joint-wrench, and ray-caster sensors without charging physics stepping to sensor timing.

Explain why isolated asset and sensor benchmarks exist, how to run each backend workload, and how to interpret and compare their outputs.

# Conflicts: # scripts/benchmarks/rl_games/benchmark_rl_games_play.py # scripts/benchmarks/rl_games/benchmark_rl_games_train.py # scripts/benchmarks/rsl_rl/benchmark_rsl_rl_play.py # scripts/benchmarks/rsl_rl/benchmark_rsl_rl_train.py # scripts/benchmarks/runtime.py # scripts/benchmarks/sb3/benchmark_sb3_play.py # scripts/benchmarks/sb3/benchmark_sb3_train.py # scripts/benchmarks/skrl/benchmark_skrl_play.py # scripts/benchmarks/skrl/benchmark_skrl_train.py

# Conflicts: # docs/source/testing/micro_benchmarks.rst # scripts/benchmarks/benchmark_load_robot.py

Make runtime and startup help bypass the required task check so the unified benchmark CLI can render its full options. Cover the forwarded-argv path that exposed the parser ordering bug.

Target the public benchmark backend modules after the standalone scripts became compatibility wrappers. This keeps parser validation attached to the implementation that owns it.

# Description PR isaac-sim#6505 removed the Kit extension startup that previously re-imported `isaaclab_physx` after Kit loaded. When task configuration imports the PhysX package before Kit, the compatibility hook returns early and is never retried. The original Isaac Sim `SimulationManager` STOP callback then invalidates the tensor view shared with `PhysxManager`, causing Franka and other PhysX articulation environments to fail during their first reset. This PR retries the existing idempotent compatibility takeover from `PhysxManager.initialize()`, where Kit is available and PhysX is the selected backend. It also updates the import-order contract and adds an integration regression test that imports `PhysxCfg` before launching Kit. This fix is independent of isaac-sim#6337. ## Type of change - Bug fix (non-breaking change which fixes an issue) - Documentation update ## Screenshots N/A ## Validation - Regression RED on unmodified develop: module alias remained the original Isaac Sim `SimulationManager` after `SimulationContext` initialization. - Regression GREEN: `1 passed`. - `Isaac-Reach-Franka`, 16 environments, PhysX: environment creation, reset, and first step completed with exit code 0 and without invalidated-view errors. - `./isaaclab.sh -p tools/changelog/cli.py check develop` - `./isaaclab.sh -f` ## Checklist - [x] I have read and understood the contribution guidelines - [x] I have run the pre-commit checks with `./isaaclab.sh -f` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove the fix is effective - [x] I have added a changelog fragment for every touched package - [x] My name already exists in `CONTRIBUTORS.md`

…-user-guide # Conflicts: # docs/source/api/lab_rl/isaaclab_rl.rst # scripts/benchmarks/play.py # scripts/benchmarks/rl_games/benchmark_rl_games_play.py # scripts/benchmarks/rl_games/benchmark_rl_games_train.py # scripts/benchmarks/rsl_rl/benchmark_rsl_rl_play.py # scripts/benchmarks/rsl_rl/benchmark_rsl_rl_train.py # scripts/benchmarks/sb3/benchmark_sb3_play.py # scripts/benchmarks/sb3/benchmark_sb3_train.py # scripts/benchmarks/skrl/benchmark_skrl_play.py # scripts/benchmarks/skrl/benchmark_skrl_train.py # scripts/benchmarks/training.py # scripts/reinforcement_learning/ray/tuner.py # source/isaaclab_rl/isaaclab_rl/entrypoints/api.py # source/isaaclab_rl/isaaclab_rl/entrypoints/dispatch.py # source/isaaclab_rl/test/test_entrypoints.py

Keep typed training and play requests equivalent to the CLI added by PR isaac-sim#6474. Translate warm-up steps through dispatch so callers do not need backend-specific arguments.

Bring the latest PR isaac-sim#6564 typed training and play request contract into the benchmark documentation stack.

Scope sys.argv to the typed request while benchmark workflows run so nested AppLauncher parsing sees the same arguments as the workflow entry point. Restore the caller arguments after dispatch.

AntoineRichard · 2026-07-20T10:42:46Z

Merged and pushed the latest #6564 typed-dispatch fix into this branch as dependency commit aa37b66c365. This keeps the Part 2 advanced-guide comparison documentation-only.

AntoineRichard and others added 30 commits July 13, 2026 09:48

Improve runtime benchmark sampling

97a00b1

Exclude configurable warmup frames from runtime metrics and compute throughput from total measured work over total wall time. Use longer defaults so the quick benchmark is less sensitive to startup and short sampling windows.

Measure simulation-step host time

8d0b99d

Add an opt-in runtime benchmark probe around the environment's SimulationContext step calls. Report host time inside and outside that boundary while preserving the original method after measurement.

Report synchronized benchmark overhead

7fb24c1

Measure environment and synchronized simulation stepping across runtime, training, and play benchmarks. Store the resulting Isaac Lab overhead in benchmark schema 1.1 while leaving regular training and play entry points uninstrumented.

Make synchronized benchmark timing optional

9521321

Require an explicit benchmark flag before wrapping environment and simulation steps. Disabled benchmarks now avoid method replacement and device synchronization while schema 1.1 continues to represent measured runs.

Rename benchmark overhead flag

ee04ef9

Name the opt-in instrumentation after the Isaac Lab overhead it reports instead of the implementation detail used to derive it.

Exclude pending work from overhead timing

4a59a05

Synchronize before environment and simulation timing begins so policy and action-generation work queued earlier is not attributed to Isaac Lab overhead.

Center throughput deviation on effective mean

ab28226

Recalculate throughput standard deviation around the aggregate effective mean so benchmark MeanStd values use one consistent center.

Reject empty benchmark workloads

e648170

Validate workload counts before launch and require positive environment timing samples so empty synchronized runs fail with actionable errors instead of division by zero.

Cover optional benchmark timing paths

7a2a118

Run the training and play smoke matrix without synchronized overhead measurement, retain one focused synchronized replay, and verify pending work before env.step is excluded from its metric.

Stabilize benchmark timing diagnostics

c292852

Keep the first runtime step outside throughput regardless of overhead instrumentation and floor derived overhead at zero when nested wall timers cross due to measurement noise.

Refactor train and play into rl subpackage

35a3777

Fix changelog fragment

2e8f35a

Merge RL entrypoint refactor dependency

b763ab5

Bring PR isaac-sim#6553 on top of the benchmark timing work from PR isaac-sim#6474 so the benchmark library can dispatch the new RL entrypoints directly.

Fix articulation data micro-benchmarks

4421ee0

Use one device-wide synchronization per timing boundary and keep mock backend state preallocated between samples. Initialize ordering and dynamics buffers so PhysX and Newton accessors are measured without hidden skips or allocation work.

Centralize deprecated benchmark aliases

63092f3

Replace per-file Python shims with lazy module proxies so legacy imports remain compatible while Git can recognize the benchmark implementation moves. Retain type forwarding stubs and document removal in Isaac Lab 3.1.

Fix OVPhysX mock structured reads

08abc71

Preserve the destination Warp dtype when copying mock binding data so structured pose and spatial-vector buffers can be exercised by asset data benchmarks.

Add OVPhysX asset micro-benchmarks

095acc4

Cover articulation, rigid object, and rigid object collection method and data access performance using kitless mock bindings comparable to the Newton benchmark matrix.

Format OVPhysX asset benchmarks

a12af09

Apply the repository Ruff format to the newly tracked benchmark scripts and keep fill-ratio selection conditions within the line-length limit.

Add OVPhysX sensor micro-benchmarks

8628973

Measure synchronized updates, submission latency, and backend reads for contact, frame-transformer, IMU, PVA, joint-wrench, and ray-caster sensors without charging physics stepping to sensor timing.

Add Newton sensor micro-benchmarks

827345e

Document micro-benchmark workflows

b9d681f

Explain why isolated asset and sensor benchmarks exist, how to run each backend workload, and how to interpret and compare their outputs.

Merge micro-benchmark documentation stack

9970c5e

# Conflicts: # docs/source/testing/micro_benchmarks.rst # scripts/benchmarks/benchmark_load_robot.py

Allow benchmark help without task

e260c36

Make runtime and startup help bypass the required task check so the unified benchmark CLI can render its full options. Cover the forwarded-argv path that exposed the parser ordering bug.

AntoineRichard added 11 commits July 17, 2026 13:47

Merge benchmark API help fix

e0fa2fa

Update benchmark smoke test imports

4067c87

Target the public benchmark backend modules after the standalone scripts became compatibility wrappers. This keeps parser validation attached to the implementation that owns it.

Merge benchmark smoke test fix

e53f067

Reorganize benchmark documentation

2493f14

Use standard benchmark output examples

fa6f833

Rewrite benchmark user guide

216ce31

Explain startup iteration cost

6031ec5

Fix benchmark schema provenance

f02ca89

Add benchmark workflow examples

64a9295

github-actions Bot added documentation Improvements or additions to documentation isaac-lab Related to Isaac Lab team labels Jul 17, 2026

AntoineRichard added 5 commits July 20, 2026 11:16

Merge commit 'fba71b97d1a' into antoiner/benchmark-library-api

ce78a6e

Expose RL benchmark warm-up in Python API

ed5eae8

Keep typed training and play requests equivalent to the CLI added by PR isaac-sim#6474. Translate warm-up steps through dispatch so callers do not need backend-specific arguments.

Merge benchmark API warm-up update

ba72445

Bring the latest PR isaac-sim#6564 typed training and play request contract into the benchmark documentation stack.

Fix typed benchmark launcher arguments

ae91002

Scope sys.argv to the typed request while benchmark workflows run so nested AppLauncher parsing sees the same arguments as the workflow entry point. Restore the caller arguments after dispatch.

Merge typed benchmark dispatch fix

aa37b66

AntoineRichard mentioned this pull request Jul 20, 2026

Rewrite advanced benchmark framework guide #6627

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Rewrite benchmark user guide#6595

Rewrite benchmark user guide#6595
AntoineRichard wants to merge 46 commits into
isaac-sim:developfrom
AntoineRichard:antoiner/benchmark-user-guide

AntoineRichard commented Jul 17, 2026

Uh oh!

AntoineRichard commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

AntoineRichard commented Jul 17, 2026

Description

Review boundary and dependencies

Workstation captures

Type of change

Screenshots

Validation

Checklist

Uh oh!

AntoineRichard commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants