Refactor viz and recording cfg classes: passive recorder, render callbacks, ViewerCfg removal#6598
Open
matthewtrepte wants to merge 9 commits into
Conversation
… Newton/Rerun/Viser)
…er accesses Fix critical crash in ManagerBasedRLEnv.__init__ where cfg.viewer.eye/lookat was accessed after the viewer field was removed from the env cfg base classes. The recorder is now passive and no longer needs camera position forwarded. Migrate all envs that silently lost their ViewerCfg without replacement: g1 locomanipulation SDG, dexsuite reorient, g129 trocar assembly, Spot flat, cartpole direct, cartpole direct-camera, and cartpole-showcase camera envs all now set sim.default_visualizer_cfg = VisualizerCfg(eye=..., lookat=...) in __post_init__, following the pattern of the other migrated tasks. Fix visualizer_integration_utils.py stale env_cfg.viewer.eye/lookat access that would crash all cartpole visualizer integration tests; migrated to env_cfg.sim.default_visualizer_cfg = VisualizerCfg(...) instead.
Export VideoRecorderCfg from isaaclab.envs so the public namespace reflects the new recorder API; the class lived in utils/ and was not reachable via lazy_export, breaking the install CI smoke test. Add five unit tests for add_render_callback / remove_render_callback in test_simulation_context.py — the new public API introduced by the re-arch had zero test coverage. Update stale docstrings in the two headless-video-pump regression tests that still referenced the deleted recording_hooks module as the implementation mechanism; the same contract is now wired through add_render_callback by the physics-backend managers.
… paths Four integration tests covering every VideoRecorder capture path: kit-visualizer-source, newton-visualizer-source, physx-renderer-source, newton-renderer-source. Each creates a CartpoleEnv with render_mode="rgb_array", steps 10 physics ticks, calls env.render(), and asserts the frame is a non-trivial (_H, _W, 3) uint8 ndarray with at least 1% non-black pixels. Previously test_record_video.py only exercised the old gymnasium RecordVideo wrapper without touching VideoRecorderCfg, and test_video_recorder.py only used mocks. These tests exercise the real capture stack end-to-end.
The previous Newton visualizer test used NewtonVisualizerCfg (abstract base, visualizer_type=None) instead of NewtonGLVisualizerCfg (visualizer_type="newton"). This made it accidentally test the physics manager fallback rather than the Newton GL visualizer direct path. Rewrite to cover all three _select_video_backend dispatch paths: - Path 1 (type="kit"): Kit visualizer with PhysX and with Newton physics (cross-backend — Kit capture is renderer-agnostic) - Path 2 (type="newton"): NewtonGLVisualizerCfg direct framebuffer capture - Path 3 (fallthrough): NewtonRTXVisualizerCfg (type="newton_rtx") is not recognised so falls through to the physics manager; also standalone renderer-source tests for PhysX and Newton backends
Three issues found by running the tests:
- Kit Replicator render products are zero-initialized and need RTX warmup
frames before the buffer populates. Add a poll-until-non-black loop
(_WARMUP_RENDER_BUDGET=40 sim.render() calls) matching the same pattern
used by OVRTX tests in rendering_test_utils.py.
- Cartpole against a black background is sparse; lower _MIN_NONZERO_RATIO
from 0.01 to 0.005 so a valid but thin scene does not false-fail.
- Kit Replicator recording produces a zero buffer when Newton is the active
physics backend. The PhysX manager registers an ensure_isaac_rtx_render_update
render callback that primes the render product; Newton does not register an
equivalent, so the buffer stays zeroed even with Kit visualizer pumping
app.update(). Mark test_kit_visualizer_newton_source_records_rgb as
strict xfail to document this known gap.
- NewtonRTXVisualizerCfg is not a factory-registered visualizer type
("newton_rtx" raises ValueError). Remove the broken fallthrough test for
it; path-3 fallthrough logic is already unit-tested in test_video_recorder.py.
Final result: 4 passed, 1 xfailed in 35s.
Newton's Fabric transform writes (via wp.fabricarray Warp kernel) do not trigger RTX's scene delegate change notifications, so Replicator annotator buffers stay zeroed regardless of how many app.update() calls are issued. Fix _select_video_backend to detect Newton physics (video_capture_backend() == "newton_gl") and skip Kit Replicator in that case. If a Newton GL visualizer is configured, use its framebuffer directly. If only a Kit visualizer is configured alongside Newton, log a warning and fall back to standalone Newton GL capture, which renders Newton's scene correctly. Kit recording continues to work unchanged with PhysX. Add two unit tests for the new fallback dispatch and remove the xfail from test_kit_visualizer_newton_source_records_rgb — all 5 integration tests now pass (previously 4 passed, 1 xfailed).
matthewtrepte
requested review from
AntoineRichard,
ClemensSchwarke,
Mayankm96,
StafaH,
Toni-SM,
aserifi,
daniela-hase,
david-cao-mueller,
frlai,
huidongc,
jaybdub,
kellyguo11,
ooctipus,
pascal-roth,
pbarejko,
peterd-NV,
r-schmitt,
rubengrandia and
vidurv-nvidia
as code owners
July 18, 2026 03:07
github-actions
Bot
added
documentation
Contributor
Greptile SummaryThis PR performs a large architectural cleanup of the visualization and recording stack, replacing
Confidence Score: 3/5The core recording pipeline and render-callback refactor are sound, but the UI camera control panel has functional regressions that affect interactive use. The recording refactor is well-tested and architecturally clean. The base_env_window.py camera UI has two regressions: the origin-type dropdown no longer repositions the viewport camera immediately for world/env modes, and eye/lookat slider changes during asset-tracking mode are silently undone on the next render step because viz.cfg.eye/lookat are not written back. source/isaaclab/isaaclab/envs/ui/base_env_window.py — both the origin-type callback and the location callback need fixes before the UI camera controls behave correctly. Important Files Changed
Sequence Diagram%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant Env as DirectRLEnv
participant SimCtx as SimulationContext
participant PhysxMgr as PhysxManager
participant KitViz as KitVisualizer
participant VidRec as VideoRecorder
Note over Env,VidRec: Initialization
Env->>SimCtx: __init__() sets _render_callbacks
SimCtx->>PhysxMgr: initialize(sim)
PhysxMgr->>SimCtx: add_render_callback(physx_headless_video_pump)
Env->>KitViz: initialize()
KitViz->>KitViz: _setup_initial_camera_view()
Env->>VidRec: VideoRecorder(cfg, scene)
Note over Env,VidRec: Per-step render loop
Env->>SimCtx: render()
SimCtx->>KitViz: step(dt)
KitViz->>KitViz: _update_asset_tracking_camera()
loop sorted _render_callbacks
SimCtx->>PhysxMgr: pump_kit_app_for_headless_video_render_if_needed
end
Env->>VidRec: render_rgb_array()
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant Env as DirectRLEnv
participant SimCtx as SimulationContext
participant PhysxMgr as PhysxManager
participant KitViz as KitVisualizer
participant VidRec as VideoRecorder
Note over Env,VidRec: Initialization
Env->>SimCtx: __init__() sets _render_callbacks
SimCtx->>PhysxMgr: initialize(sim)
PhysxMgr->>SimCtx: add_render_callback(physx_headless_video_pump)
Env->>KitViz: initialize()
KitViz->>KitViz: _setup_initial_camera_view()
Env->>VidRec: VideoRecorder(cfg, scene)
Note over Env,VidRec: Per-step render loop
Env->>SimCtx: render()
SimCtx->>KitViz: step(dt)
KitViz->>KitViz: _update_asset_tracking_camera()
loop sorted _render_callbacks
SimCtx->>PhysxMgr: pump_kit_app_for_headless_video_render_if_needed
end
Env->>VidRec: render_rgb_array()
Reviews (1): Last reviewed commit: "Fix Kit+Newton recording gap: skip Kit R..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
393
to
+407
| def _set_viewer_origin_type_fn(self, value: str): | ||
| """Sets the origin of the viewport's camera. This is based on the drop-down menu in the UI.""" | ||
| # Extract the viewport camera controller from environment | ||
| vcc = self.env.viewport_camera_controller | ||
| if vcc is None: | ||
| raise ValueError("Viewport camera controller is not initialized! Please check the rendering mode.") | ||
| viz = self._get_kit_visualizer() | ||
| if viz is None: | ||
| return | ||
|
|
||
| # Based on origin type, update the camera view | ||
| if value == "World": | ||
| vcc.update_view_to_world() | ||
| viz.cfg.origin_type = "world" | ||
| elif value == "Env": | ||
| vcc.update_view_to_env() | ||
| viz.cfg.origin_type = "env" | ||
| else: | ||
| # find which index the asset is | ||
| fancy_names = [name.replace("_", " ").title() for name in self._viewer_assets_options] | ||
| # store the desired env index | ||
| viewer_asset_name = self._viewer_assets_options[fancy_names.index(value)] | ||
| # update the camera view | ||
| vcc.update_view_to_asset_root(viewer_asset_name) | ||
| viz.cfg.origin_type = "asset_root" | ||
| viz.cfg.asset_name = viewer_asset_name |
Contributor
There was a problem hiding this comment.
Changing to "World" or "Env" now only mutates viz.cfg.origin_type but never calls _setup_initial_camera_view() or _apply_viewer_origin_to_camera(), so the viewport camera stays frozen at its previous position. The old ViewportCameraController.update_view_to_world() / update_view_to_env() immediately computed the new viewer_origin and called update_view_location(). The same is true for the env-index spinner in _set_viewer_env_index_fn (line 418): setting viz.cfg.env_index has no effect until the next asset-tracking step because KitVisualizer.step() only updates the camera for "asset_root" / "asset_body" origins.
Each branch should additionally call the camera-setup helpers (or a new public method that re-runs the origin resolution and applies it) so that the dropdown and spinner produce the same immediate repositioning users expected before.
Comment on lines
409
to
+416
| def _set_viewer_location_fn(self, model: omni.ui.SimpleFloatModel): | ||
| """Sets the viewport camera location based on the UI.""" | ||
| # access the viewport camera controller (for brevity) | ||
| vcc = self.env.viewport_camera_controller | ||
| if vcc is None: | ||
| raise ValueError("Viewport camera controller is not initialized! Please check the rendering mode.") | ||
| # obtain the camera locations and set them in the viewpoint camera controller | ||
| viz = self._get_kit_visualizer() | ||
| if viz is None: | ||
| return | ||
| eye = [self.ui_window_elements["viewer_eye"][i].get_value_as_float() for i in range(3)] | ||
| lookat = [self.ui_window_elements["viewer_lookat"][i].get_value_as_float() for i in range(3)] | ||
| # update the camera view | ||
| vcc.update_view_location(eye, lookat) | ||
| viz.set_camera_view(eye, lookat) |
Contributor
There was a problem hiding this comment.
viz.set_camera_view(eye, lookat) moves the viewport camera to the given absolute position, but viz.cfg.eye and viz.cfg.lookat (the relative offsets used by _apply_viewer_origin_to_camera) are never updated. On the very next call to KitVisualizer.step() in "asset_root" or "asset_body" mode, _update_asset_tracking_camera() recomputes the position from the old cfg.eye/cfg.lookat and overwrites whatever the user just set.
The old ViewportCameraController.update_view_location() avoided this by storing the new values in self.default_cam_eye / self.default_cam_lookat before applying them. The fix is to also write back: viz.cfg.eye = tuple(eye); viz.cfg.lookat = tuple(lookat) before calling set_camera_view.
Comment on lines
30
to
+31
| from . import mdp, ui | ||
| from .common import VecEnvObs, VecEnvStepReturn, ViewerCfg | ||
| from .common import VecEnvObs, VecEnvStepReturn |
Contributor
There was a problem hiding this comment.
VideoRecorderCfg is now exported from __init__.py (the test file test_recording_backends.py imports it from isaaclab.envs), but it is absent from the .pyi type stub. Type-checkers and IDE completions will not see it.
Suggested change
| from . import mdp, ui | |
| from .common import VecEnvObs, VecEnvStepReturn, ViewerCfg | |
| from .common import VecEnvObs, VecEnvStepReturn | |
| from . import mdp, ui | |
| from .common import VecEnvObs, VecEnvStepReturn | |
| from .utils.video_recorder_cfg import VideoRecorderCfg |
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
…ure classes
Remove IsaacsimKitPerspectiveVideo/Cfg and NewtonGlPerspectiveVideo/Cfg.
Recording is now fully internal to env.step() — no gym RecordVideo wrapper
needed. VideoRecorderCfg is backend-agnostic and specifies only what to
record and where to write it, not how to capture.
VideoRecorderCfg fields:
source: str = "visualizer" source string (visualizer[:<type>[/tiled]],
sensor:<name>[/<data_type>])
output_dir: str clip output directory
fps: int output video frame rate
clip_length: int env steps per clip
clip_trigger_step: int start a new clip every N steps (0 = once)
env cfgs gain video_recorders: list[VideoRecorderCfg] = [] (empty = off)
replacing the single video_recorder: VideoRecorderCfg field. Multiple
entries produce independent simultaneous streams from different sources.
KitVisualizer gains render_rgb_array() via a lazily-created Replicator
annotator on its controlled camera prim. Newton GL visualizer already had
render_rgb_array(). Sensor sources read from env.scene.sensors[name].
VideoRecorder.step() is called in DirectRLEnv, ManagerBasedRLEnv, and
DirectMARLEnv step loops after physics/render and before observations.
|
This looks like it'll make simulation control so much smoother! 🗺️✨😌 Maintainer? Turn off weaves from non-maintainers → |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR replaces the
ViewerCfg/ViewportCameraController/recording_hooksstack with a cleaner, backend-agnostic architecture:ViewerCfgfrom all env cfg bases (DirectRLEnvCfg,ManagerBasedEnvCfg,DirectMARLEnvCfg) and migrates all task envs tosim.default_visualizer_cfg = VisualizerCfg(eye=..., lookat=...)orsim.visualizer_cfgs = [KitVisualizerCfg(...)]ViewportCameraController(243 lines) — camera tracking is now handled directly byKitVisualizervia itsorigin_type/asset_name/body_namecfg fieldsrecording_hooksmodule from bothisaaclabandisaaclab_newton— physics-backend recording callbacks are now registered via the newSimulationContext.add_render_callback(name, fn, order)public APIVideoRecorder:VideoRecorderCfg.eye/lookatremoved; the recorder captures whatever the active visualizer renders without repositioning the cameraPhysicsManager.video_capture_backend(): new classmethod replaces name-sniffing heuristic for backend selectionVideoRecorderCfgexported fromisaaclab.envspublic namespace_select_video_backendskips Kit Replicator (Newton Fabric writes don't notify RTX's scene delegate) and falls back to standalone Newton GL capture with a logged warningNet diff: 69 files changed, +750 / −745 (≈ net zero lines despite the significant cleanup, because new
KitVisualizercamera tracking code replaces the deletedViewportCameraController).Test plan
source/isaaclab/test/envs/test_video_recorder.py— 17/17 unit tests pass (covers all_select_video_backenddispatch paths including new Newton fallback)source/isaaclab/test/sim/test_simulation_context.py— 29 passed before pre-existing hang ontest_timeline_callbacks_with_weakref(unrelated)source/isaaclab/test/envs/test_env_rendering_logic.py— 17/17 passsource/isaaclab/test/sim/test_simulation_context_visualizers.py— 28/28 passsource/isaaclab_tasks/test/core/test_recording_backends.py— 5/5 integration tests pass (Kit/PhysX, Kit/Newton fallback, Newton GL, PhysX renderer, Newton renderer)🤖 Generated with Claude Code