[Bug Report] Teleoperation example in IsaacLab not working with cloudxr-runtime:5.0.0

[errrr0501]

If you are submitting a bug report, please fill in the following details and use the tag [bug].

Describe the bug

When performing teleoperation in IsaacLab using GR1-T2 manipulation, the robot does not move as expected when running with docker cloudxr-runtime:5.0.0. The same teleoperation setup works correctly with the older version docker cloudxr-runtime:0.1.0-isaac.

This behavior seems to be specific to the new runtime, as no other parts of the teleoperation setup were changed between tests.

Steps to reproduce

1. Launch IsaacLab with GR1-T2 teleoperation task.

2. Set up teleoperation (same configuration as with cloudxr-runtime:0.1.0-isaac).

3. Use cloudxr-runtime:5.0.0 for streaming.

Describe the characteristic of your environment:

• Commit: 280b667 (branch: main)
• Isaac Sim Version: 5.0.0
• OS: Ubuntu 22.04
• GPU: RTX 5090
• CUDA: 12.8
• GPU Driver: 570.169

Additional context

Add any other context about the problem here.

Checklist

• [*] I have checked that there is no similar issue in the repo (required)
• [*] I have checked that the issue is not in running Isaac Sim itself and is related to the repo

[rwiltz]

Hi @errrr0501, it sounds like maybe you have not upgraded to the corresponding 2.2 version of the IsaacXRTeleop client app. Are you able to try the latest app from here and report back?

[errrr0501]

Hi @rwiltz , thanks for the quick response!
I updated to the IsaacXRTeleop client app v2.2 as suggested, and the GR1-T2 teleoperation now works as expected with cloudxr-runtime:5.0.0.

However, I noticed a separate issue:
• I’m using the same hand model as before, but put it on different manipulator.
• When I move my left finger in the real world, the right finger moves in the simulator (and vice versa).
• Wrist movements are correct, so it seems isolated to finger mapping.

Could this be caused by the retargeting or by my hand joint mapping settings?(although I follow the GR1-T2 setting) If you have guidance on where to check or adjust this mapping, I can test further.

Appreciate the help!

[rwiltz]

Hi @errrr0501, the order of the joint angles in the tensor that is passed to the env via env.step is determined by the order of the joint list in the device cfg Example:

self.teleop_devices = DevicesCfg(
            devices={
                "handtracking": OpenXRDeviceCfg(
                    retargeters=[
                        GR1T2RetargeterCfg(
                            enable_visualization=True,
                            # OpenXR hand tracking has 26 joints per hand
                            num_open_xr_hand_joints=2 * 26,
                            sim_device=self.sim.device,
                            #This list determines the order of the joints
                            hand_joint_names=self.actions.pink_ik_cfg.hand_joint_names,
                        ),
                    ],
                    sim_device=self.sim.device,
                    xr_cfg=self.xr,
                ),
            }

From here

[errrr0501]

Hi @rwiltz, thanks for clarifying how the joint list determines the order of the tensor.

I did double-check and followed the same joint order in my config, but the issue still occurs:

Moving my left hand in the real world drives the right hand joints in the simulator (and vice versa).

Wrist mappings are correct — the reversal only happens for the fingers.

As a temporary workaround, I remapped the captured right-hand joints to the simulated left hand and the captured left-hand joints to the simulated right hand, which resolves the mirroring issue for now.

# Compute per-hand joint angles from the tracker
left_hands_pos = self._hands_controller.compute_left(left_hand_poses)
right_hands_pos = self._hands_controller.compute_right(right_hand_poses)

# Build index lists using the configured joint-name order
left_idx = [self._hand_joint_names.index(n) for n in self._hands_controller.get_left_joint_names()]
right_idx = [self._hand_joint_names.index(n) for n in self._hands_controller.get_right_joint_names()]

# Retarget buffers
n = len(self._hands_controller.get_joint_names())
left_retargeted_hand_joints = np.zeros(n)
right_retargeted_hand_joints = np.zeros(n)

# --- Workaround: swap hands ---
left_retargeted_hand_joints[left_idx] = right_hands_pos   # right → left
right_retargeted_hand_joints[right_idx] = left_hands_pos  # left → right

# Outputs used by the env
left_hand_joints = left_retargeted_hand_joints
right_hand_joints = right_retargeted_hand_joints

Do you have suggestions on where to properly adjust this so I don’t need to flip them manually?

[RandomOakForest]

Following up on this, I will move this post to our Discussions section for follow up. If a bug is documented, please open a new bug report with an update. Thank you.