Two Arm Environments for Whole_BODY_IK composite controller does not correctly move the second arm

[Farbod82]

System Info

Robosuite: 1.5.1 OS: Windows

Information

Hi,

import numpy as np
import robosuite as suite
from robosuite import load_composite_controller_config


controls = load_composite_controller_config(controller="WHOLE_BODY_IK")


config = {
        "env_name": "TwoArmLift",
        "robots": ["Panda","Panda"],
        "controller_configs": controls,
        "env_configuration":"parallel" 
}

env = suite.make(
    **config,
    has_renderer=True,
    has_offscreen_renderer=True,
    ignore_done=True,
    use_camera_obs=True,
    reward_shaping=True,
    control_freq=20,
)
obs1 = env.reset()

for i in range(100):
    action = np.zeros(env.action_dim)
    action[0] = obs1["robot0_eef_pos"][0]
    action[1] = obs1["robot0_eef_pos"][1]
    action[2] = obs1["robot0_eef_pos"][2]
    action[3] = -np.pi/2
    action[4] = 0
    action[5] = 0
    action[7] = obs1["robot1_eef_pos"][0]
    action[8] = obs1["robot1_eef_pos"][1]
    action[9] = obs1["robot1_eef_pos"][2]
    action[10] = -np.pi/2
    action[11] = 0
    action[12] = 0
 
    obs, reward, done, _ = env.step(action)

env.close()

The first arm functions correctly(robot0) but the second arm(robot1) always goes up and ignores the given actions.
I want to create a bimanual env with two Panda robots that I can give absolute position and orientations.
The env works well in BASIC control mode with delta config.
I would appreciate any help.
Thank you.

Reproduction

import numpy as np
import robosuite as suite
from robosuite import load_composite_controller_config


controls = load_composite_controller_config(controller="WHOLE_BODY_IK")


config = {
        "env_name": "TwoArmLift",
        "robots": ["Panda","Panda"],
        "controller_configs": controls,
        "env_configuration":"parallel" 
}

env = suite.make(
    **config,
    has_renderer=True,
    has_offscreen_renderer=True,
    ignore_done=True,
    use_camera_obs=True,
    reward_shaping=True,
    control_freq=20,
)
obs1 = env.reset()

for i in range(100):
    action = np.zeros(env.action_dim)
    action[0] = obs1["robot0_eef_pos"][0]
    action[1] = obs1["robot0_eef_pos"][1]
    action[2] = obs1["robot0_eef_pos"][2]
    action[3] = -np.pi/2
    action[4] = 0
    action[5] = 0
    action[7] = obs1["robot1_eef_pos"][0]
    action[8] = obs1["robot1_eef_pos"][1]
    action[9] = obs1["robot1_eef_pos"][2]
    action[10] = -np.pi/2
    action[11] = 0
    action[12] = 0
 
    obs, reward, done, _ = env.step(action)

env.close()

Expected behavior

No response

[Abhiram824]

Hello, thanks for bringing up this issue. I've created a branch (abs_action_multirobotenv_fix) for the time being that should fix the problem you are observing: could you check it out and let me know if the issue is gone?

[Farbod82]

Hi, thanks for your answer,
unfortunately I think the issue still exists.
I debugged the code and I think the problem is here:
line 435, composite_controller.py

  original_ref_names = self.composite_controller_specific_config.get("ref_name", [])
  original_ref_names = [name.format(idn=self.robot_model.idn) for name in original_ref_names]

the references here are both for robot0_grip_left and robot0_grip_right but the mujoco reference in the simulation for this is robot0_grip_right and robot1_grip_right.
I think there is a naming inconsistency in referencing the second arm in two arm envs between the controller and mujoco.

[Abhiram824]

Ah I see, I wasnt running your test, script. Can you try running collect_human_demonstrations.py with the the same setup or running the following version of your modified test script. Make sure to stay on the same branch i created

import numpy as np
import robosuite as suite
from robosuite import load_composite_controller_config
import copy


controls = load_composite_controller_config(controller="WHOLE_BODY_IK")
controls = [controls, copy.deepcopy(controls)]


config = {
        "env_name": "TwoArmLift",
        "robots": ["Panda","Panda"],
        "controller_configs": controls,
        "env_configuration":"parallel" 
}

env = suite.make(
    **config,
    has_renderer=True,
    has_offscreen_renderer=True,
    ignore_done=True,
    use_camera_obs=True,
    reward_shaping=True,
    control_freq=20,
)
obs1 = env.reset()

for i in range(100):
    action = np.zeros(env.action_dim)
    action[0] = obs1["robot0_eef_pos"][0]
    action[1] = obs1["robot0_eef_pos"][1]
    action[2] = obs1["robot0_eef_pos"][2]
    action[3] = -np.pi/2
    action[4] = 0
    action[5] = 0
    action[7] = obs1["robot1_eef_pos"][0]
    action[8] = obs1["robot1_eef_pos"][1]
    action[9] = obs1["robot1_eef_pos"][2]
    action[10] = -np.pi/2
    action[11] = 0
    action[12] = 0
 
    obs, reward, done, _ = env.step(action)

env.close()

[Farbod82]

Thanks, it works now. So I should have duplicated the controller. I thought one composite controller containing left and right body_parts was enough. Thank you for the help.