Add optional teleop tracking debug visualization

[hougantc-nvda]

Description

Adds optional tracking debug visualization for IsaacTeleop sessions (reviving the concept from #4840 on the current stack):

• Hand joints — red spheres at each OpenXR hand joint (26/hand), when the pipeline contains a HandsSource.
• Controllers — RGB axis markers at each controller's OpenXR aim pose.

Key points:

• Opt-in, zero overhead when off. Enabled per-run via the enable_debug_visualization argument of create_isaac_teleop_device() (exposed as --enable_debug_visualization on teleop_se3_agent.py, record_demos.py, and teleop_replay_agent.py). Disabled sessions build an identical pipeline; env cfgs are untouched.
• Startup-only. The debug outputs are chained at session start; there is no mid-session toggle.
• Frame-correct. Raw anchor-frame tracking data is transformed to world on the Isaac Lab side using the anchor manager's world_T_anchor, so markers stay correct under a target_T_world rebase (fixing the original Adds back optional Teleop hand joint visualization #4840 approach).

Also hardens teleop step-failure handling: a retargeting pipeline error is now distinguished from an external XR teardown (via Kit's XR session state), and session re-creation is rate-limited to once per second, preventing per-frame restart churn on degenerate tracking data. Stop-AR/Start-AR recovery latency is unchanged.

Validated live over CloudXR (Quest 3).

Type of change

• New feature (non-breaking change which adds functionality)

Checklist

• I have read and understood the contribution guidelines
• I have run the pre-commit checks with ./isaaclab.sh --format
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• I have updated the changelog (via a changelog.d fragment)
• I have added my name to the CONTRIBUTORS.md or my name already exists there

[greptile-apps]

Greptile Summary

This PR adds optional tracking debug visualization to IsaacTeleop sessions — red sphere markers at OpenXR hand joints and RGB axis markers at controller aim poses — and also hardens step-failure handling with proper failure diagnosis and a restart rate-limit.

• Debug visualization pipeline: new HandJointVisualizer and ControllerAimVisualizer classes are chained into the OutputCombiner only when enabled, keeping zero overhead by default; a toggle_debug_visualization control message can lazily enable or show/hide markers at runtime via an in-place session restart.
• Frame-correct transforms: raw anchor-frame tracking data is transformed to world frame using the anchor manager's world_T_anchor directly (fixing a bug from the earlier Adds back optional Teleop hand joint visualization #4840 approach where the rebased pipeline transform was reused).
• Step-failure hardening: failures are now diagnosed against the actual Kit XR session state (pipeline error vs. external XR teardown) and session re-creation is rate-limited to once per second, preventing restart churn on persistent pipeline errors such as degenerate tracking data.

Confidence Score: 4/5

Safe to merge; all findings are display-only or documentation-level concerns in new debug visualization code, with no risk to the primary teleop pipeline.

The core pipeline change — chaining debug outputs into the OutputCombiner only when enabled — is well-isolated and properly guarded. The step-failure hardening is straightforward and tested. The main concern worth addressing before merge is that enable_debug_viz() does not persist its effect across a stop()/start() cycle: _enable_debug_visualization stays False, so a subsequent start() quietly rebuilds the pipeline without debug outputs while _viz_visible on the device remains True. The two docstring typos (hand_debug_chained and enable_hand_debug) are minor but could confuse future contributors. The ghost-marker behaviour on controller count changes is a display artifact in debug tooling only.

session_lifecycle.py around enable_debug_viz (runtime-enable persistence) and the last_step_result property docstring; test_hand_debug_viz.py module docstring.

Important Files Changed

Filename	Overview
source/isaaclab_teleop/isaaclab_teleop/session_lifecycle.py	Core lifecycle changes: new _build_combined_pipeline/_chain_hand_debug_outputs helpers, enable_debug_viz runtime restart, last_step_result/last_left_controller properties, step-failure cooldown, and consume_visualization_toggle delegation. One docstring references a non-existent attribute (hand_debug_chained) and _enable_debug_visualization is not updated in enable_debug_viz, causing debug viz to be silently lost after a stop/start cycle.
source/isaaclab_teleop/isaaclab_teleop/isaac_teleop_device.py	Adds _update_hand_debug and _handle_viz_toggle methods for lazy visualizer creation and toggle handling; integrates enable_debug_visualization argument through to the lifecycle. Logic is clean and consistent with the zero-overhead design goal.
source/isaaclab_teleop/isaaclab_teleop/visualizers/controller_aim_visualizer.py	New file: _aim_world_pose helper and ControllerAimVisualizer class. When the active controller count drops between frames the visualizer calls visualize() with fewer entries; previously-placed markers may ghost at their last position.
source/isaaclab_teleop/isaaclab_teleop/visualizers/hand_joint_visualizer.py	New file: _extract_world_joint_positions helper and HandJointVisualizer class. Anchor-to-world transform is correctly applied on the Isaac Lab side. Lazy import pattern and error latching are well-implemented.
source/isaaclab_teleop/isaaclab_teleop/teleop_message_processor.py	Adds toggle_debug_visualization command pattern and _viz_toggle_pending flag with consume_visualization_toggle; documented race is intentional and acceptable for human-scale input.
source/isaaclab_teleop/test/test_hand_debug_viz.py	New comprehensive test file; module docstring references enable_hand_debug but the actual method is enable_debug_viz.
source/isaaclab_teleop/test/test_session_step_failure.py	New test file covering step-failure diagnosis and restart holdoff; well-structured with clear test cases for both pipeline-error and XR-teardown paths.
source/isaaclab_teleop/test/test_control_events.py	Adds TestToggleVizCommand class with good coverage of the new command: plain text, JSON format, pipeline output isolation, batch collapse, word-boundary matching, and priority ordering.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Client as Script / advance()
    participant Device as IsaacTeleopDevice
    participant Lifecycle as TeleopSessionLifecycle
    participant MsgProc as TeleopMessageProcessor
    participant HViz as HandJointVisualizer
    participant AViz as ControllerAimVisualizer

    Client->>Device: advance()
    Device->>Lifecycle: step()
    Lifecycle-->>Device: action tensor
    Device->>Device: _update_hand_debug()
    Device->>Lifecycle: consume_visualization_toggle()
    alt toggle_debug_visualization received
        Lifecycle->>MsgProc: consume_visualization_toggle()
        MsgProc-->>Lifecycle: True
        Lifecycle-->>Device: True
        alt debug_viz already chained
            Device->>HViz: set_visible(toggle)
            Device->>AViz: set_visible(toggle)
        else first-time enable
            Device->>Lifecycle: enable_debug_viz()
            Lifecycle->>Lifecycle: "_build_combined_pipeline(chain_debug_viz=True)"
            Lifecycle->>Lifecycle: session.__exit__() + _try_start_session()
            Lifecycle-->>Device: True
        end
    end
    alt "_viz_visible == True"
        Device->>Lifecycle: last_step_result / last_left_controller
        Device->>HViz: update(result, world_T_anchor)
        Device->>AViz: update(left, right, world_T_anchor)
    end
    Device-->>Client: action tensor

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Client as Script / advance()
    participant Device as IsaacTeleopDevice
    participant Lifecycle as TeleopSessionLifecycle
    participant MsgProc as TeleopMessageProcessor
    participant HViz as HandJointVisualizer
    participant AViz as ControllerAimVisualizer

    Client->>Device: advance()
    Device->>Lifecycle: step()
    Lifecycle-->>Device: action tensor
    Device->>Device: _update_hand_debug()
    Device->>Lifecycle: consume_visualization_toggle()
    alt toggle_debug_visualization received
        Lifecycle->>MsgProc: consume_visualization_toggle()
        MsgProc-->>Lifecycle: True
        Lifecycle-->>Device: True
        alt debug_viz already chained
            Device->>HViz: set_visible(toggle)
            Device->>AViz: set_visible(toggle)
        else first-time enable
            Device->>Lifecycle: enable_debug_viz()
            Lifecycle->>Lifecycle: "_build_combined_pipeline(chain_debug_viz=True)"
            Lifecycle->>Lifecycle: session.__exit__() + _try_start_session()
            Lifecycle-->>Device: True
        end
    end
    alt "_viz_visible == True"
        Device->>Lifecycle: last_step_result / last_left_controller
        Device->>HViz: update(result, world_T_anchor)
        Device->>AViz: update(left, right, world_T_anchor)
    end
    Device-->>Client: action tensor

Loading

_{Reviews (1): Last reviewed commit: "Add optional teleop tracking debug visua..." | Re-trigger Greptile}