Imported MJCF not working when spawning multiple istances

[riccardofeingold]

Hi,

I've imported a mjcf description of a robot using the conversion scripts given on the IsaacLab repo with the --import-sites and --make-instanceable flags. But when I want to load multiple envs with this robot, only env_0 starts to perform the commanded movements, but the others are frozen. Here's the link to the mjcf model that I am using: https://github.com/srl-ethz/faive_gym_oss/tree/main/assets/faive_hand_p0. I also tried other more simplified mjcf files and converted it to usd, but the same problem occurs there.

Has someone an idea how to fix this issue?
Thanks in advance!

[RandomOakForest]

Thanks for posting this. Have you reviewed the doc on instanceable assets and ensured that the robot is in instanceable format? A more detailed review of this can be found in this doc.

[Yasu31]

I also have the same issue... I don't think the problem is because it's not an instanceable asset, as I've set it to be instanceable when I convert it. I have a minimal reproducible environment on a branch in my fork, for a minimal simple gripper MJCF model.

First I convert the MJCF XML to USD with:

python scripts/tools/convert_mjcf.py simple_gripper.xml simple_gripper.usd --fix-base --import-sites --make-instanceable

I added the --make-instanceable flag, so it should be instanceable already.

Then I've adapted hands.py and the Isaac-Repose-Cube-Shadow-Direct-v0 environment to load my simple_gripper. You can review my exact changes here: main...Yasu31:IsaacLab:simple-mjcf-gripper .

When I run hands.py, the gripper articulates correctly as expected:

However, when the training script is run (e.g. python scripts/reinforcement_learning/rsl_rl/train.py --task Isaac-Repose-Cube-Shadow-Direct-v0 --num_envs 32, the gripper is only articulated for the first environment:

[riccardofeingold]

Possible Solution to Fix this Issue

1. Flatten and Open in Isaac Sim
   Save the converted model as a flattened USD file and open it in Isaac Sim. This allows you to make changes easily without encountering warnings like "reparenting/moving not possible."

2. Make All Instanceable Prims Non-Instanceable
   You need to convert all instanceable prims to non-instanceable. You can either do this manually or run the following function:

def undo_instanceable_xforms(asset_usd_path, source_prim_path, save_as_path=None):
    omni.usd.get_context().open_stage(asset_usd_path)
    stage = omni.usd.get_context().get_stage()
    prims = [stage.GetPrimAtPath(source_prim_path)]

    while len(prims) > 0:
        prim = prims.pop(0)
        print(prim)
        if prim.IsInstance():
            print("This is instanceable!")
            prim.SetInstanceable(False)
        
        children_prims = prim.GetChildren()
        prims = prims + children_prims

    if save_as_path is None:
        omni.usd.get_context().save_stage()
    else:
        omni.usd.get_context().save_as_stage(save_as_path)

3. Ensure Correct Hierarchy Structure
   Make sure you follow the hierarchy structure required for proper instancing as described here:
   
   The Isaac Sim docs provide a script to do this, but it currently doesn’t work as intended. It creates the parent XForms inside the visuals and collisions XForms, but it fails to reparent the meshes into the newly created XForms.
   If anyone knows why this doesn't work, please let me know.
   For now, you'll need to manually move the meshes. Here’s the provided (non-working) script:

# TODO: Currently not adding the geometry to the newly created Xform.
def create_parent_xforms(asset_usd_path, source_prim_path, save_as_path=None):

    """ Adds a new UsdGeom.Xform prim for each Mesh/Geometry prim under source_prim_path.
        Moves material assignment to new parent prim if any exists on the Mesh/Geometry prim.
        Args:
            asset_usd_path (str): USD file path for asset
            source_prim_path (str): USD path of root prim
            save_as_path (str): USD file path for modified USD stage. Defaults to None, will save in same file.
    """
    omni.usd.get_context().open_stage(asset_usd_path)
    stage = omni.usd.get_context().get_stage()
    prims = [stage.GetPrimAtPath(source_prim_path)]
    # edits = Usd.NamespaceEditor(stage)
    edits = Sdf.BatchNamespaceEdit()
    while len(prims) > 0:
        prim = prims.pop(0)
        print(prim)
        if prim.GetTypeName() in ["Mesh", "Capsule", "Sphere", "Box", "Cone", "Torus", "Plane", "Cylinder", "Disk"]:
            new_xform = UsdGeom.Xform.Define(stage, str(prim.GetPath()) + "_xform")
            print(prim, new_xform)
            edits.Add(Sdf.NamespaceEdit.Reparent(prim.GetPath(), new_xform.GetPath(), 4))
            # edits.MovePrimAtPath(prim.GetPath(), str(new_xform.GetPath()) + str(prim.GetName()))
            # edits.ApplyEdits()
            continue
        children_prims = prim.GetChildren()
        prims = prims + children_prims
    stage.GetRootLayer().Apply(edits)

    if save_as_path is None:
        omni.usd.get_context().save_stage()
    else:
        omni.usd.get_context().save_as_stage(save_as_path)

4. Make XForms Instanceable for Efficient Cloning
   Once the parent XForms are correctly set, you can make them instanceable and add a reference to a cloned version of the model. Use the following function from the documentation:

def convert_asset_instanceable(asset_usd_path, source_prim_path, save_as_path=None, create_xforms=False):
    """ Makes all mesh/geometry prims instanceable.
        Can optionally add UsdGeom.Xform prim as parent for all mesh/geometry prims.
        Makes a copy of the asset USD file, which will be used for referencing.
        Updates asset file to convert all parent prims of mesh/geometry prims to reference cloned USD file.

        Args:
            asset_usd_path (str): USD file path for asset
            source_prim_path (str): USD path of root prim
            save_as_path (str): USD file path for modified USD stage. Defaults to None, will save in same file.
            create_xforms (bool): Whether to add new UsdGeom.Xform prims to mesh/geometry prims.
    """

    undo_instanceable_xforms(asset_usd_path, source_prim_path, save_as_path)
    asset_usd_path = save_as_path

    if create_xforms:
        create_parent_xforms(asset_usd_path, source_prim_path, save_as_path)
        asset_usd_path = save_as_path

    instance_usd_path = ".".join(asset_usd_path.split(".")[:-1]) + "_meshes.usd"
    omni.client.copy(asset_usd_path, instance_usd_path)
    omni.usd.get_context().open_stage(asset_usd_path)
    stage = omni.usd.get_context().get_stage() 

    prims = [stage.GetPrimAtPath(source_prim_path)]
    while len(prims) > 0:
        prim = prims.pop(0)
        if prim:
            if prim.GetTypeName() in ["Mesh", "Capsule", "Sphere", "Cube", "Cone", "Cylinder", "Disk", "Plane", "Torus"]:
                parent_prim = prim.GetParent()
                if parent_prim and not parent_prim.IsInstance():
                     parent_prim.GetReferences().AddReference(assetPath=instance_usd_path, primPath=str(parent_prim.GetPath()))
                     parent_prim.SetInstanceable(True)
                     continue

            children_prims = prim.GetChildren()
            prims = prims + children_prims

    if save_as_path is None:
        omni.usd.get_context().save_stage()
    else:
        omni.usd.get_context().save_as_stage(save_as_path)

Final Note
Since reparenting currently doesn’t work as expected using Sdf.NamespaceEdit.Reparent(), I’m temporarily running this script (in the script editor of IsaacSim), which makes the meshes themselves instanceable instead of their parent XForms. So, far I did not notice a difference in memory efficiency between instancing the mesh and instancing the parent XForms of the mesh. But feel free to correct me on that:

from pxr import Usd, UsdGeom, Sdf, UsdShade
import omni

# TODO: Currently not adding the geometry to the newly created Xform.
def create_parent_xforms(asset_usd_path, source_prim_path, save_as_path=None):

    """ Adds a new UsdGeom.Xform prim for each Mesh/Geometry prim under source_prim_path.
        Moves material assignment to new parent prim if any exists on the Mesh/Geometry prim.
        Args:
            asset_usd_path (str): USD file path for asset
            source_prim_path (str): USD path of root prim
            save_as_path (str): USD file path for modified USD stage. Defaults to None, will save in same file.
    """
    omni.usd.get_context().open_stage(asset_usd_path)
    stage = omni.usd.get_context().get_stage()
    prims = [stage.GetPrimAtPath(source_prim_path)]
    # edits = Usd.NamespaceEditor(stage)
    edits = Sdf.BatchNamespaceEdit()
    while len(prims) > 0:
        prim = prims.pop(0)
        print(prim)
        if prim.GetTypeName() in ["Mesh", "Capsule", "Sphere", "Box", "Cone", "Torus", "Plane", "Cylinder", "Disk"]:
            new_xform = UsdGeom.Xform.Define(stage, str(prim.GetPath()) + "_xform")
            print(prim, new_xform)
            edits.Add(Sdf.NamespaceEdit.Reparent(prim.GetPath(), new_xform.GetPath(), 4))
            # edits.MovePrimAtPath(prim.GetPath(), str(new_xform.GetPath()) + str(prim.GetName()))
            # edits.ApplyEdits()
            continue
        children_prims = prim.GetChildren()
        prims = prims + children_prims
    stage.GetRootLayer().Apply(edits)

    if save_as_path is None:
        omni.usd.get_context().save_stage()
    else:
        omni.usd.get_context().save_as_stage(save_as_path)
    # if save_as_path is None: 
    #     stage.GetRootLayer().Save()
    # else:
    #     stage.GetRootLayer().Export(save_as_path)

def undo_instanceable_xforms(asset_usd_path, source_prim_path, save_as_path=None):
    omni.usd.get_context().open_stage(asset_usd_path)
    stage = omni.usd.get_context().get_stage()
    prims = [stage.GetPrimAtPath(source_prim_path)]

    while len(prims) > 0:
        prim = prims.pop(0)
        print(prim)
        if prim.IsInstance():
            print("This is instanceable!")
            prim.SetInstanceable(False)
        
        children_prims = prim.GetChildren()
        prims = prims + children_prims

    if save_as_path is None:
        omni.usd.get_context().save_stage()
    else:
        omni.usd.get_context().save_as_stage(save_as_path)
    # if save_as_path is None: 
    #     stage.GetRootLayer().Save()
    # else:
    #     stage.GetRootLayer().Export(save_as_path)

def convert_asset_instanceable(asset_usd_path, source_prim_path, save_as_path=None, create_xforms=False):
    """ Makes all mesh/geometry prims instanceable.
        Can optionally add UsdGeom.Xform prim as parent for all mesh/geometry prims.
        Makes a copy of the asset USD file, which will be used for referencing.
        Updates asset file to convert all parent prims of mesh/geometry prims to reference cloned USD file.

        Args:
            asset_usd_path (str): USD file path for asset
            source_prim_path (str): USD path of root prim
            save_as_path (str): USD file path for modified USD stage. Defaults to None, will save in same file.
            create_xforms (bool): Whether to add new UsdGeom.Xform prims to mesh/geometry prims.
    """

    undo_instanceable_xforms(asset_usd_path, source_prim_path, save_as_path)
    asset_usd_path = save_as_path

    if create_xforms:
        create_parent_xforms(asset_usd_path, source_prim_path, save_as_path)
        asset_usd_path = save_as_path

    instance_usd_path = ".".join(asset_usd_path.split(".")[:-1]) + "_meshes.usd"
    omni.client.copy(asset_usd_path, instance_usd_path)
    omni.usd.get_context().open_stage(asset_usd_path)
    stage = omni.usd.get_context().get_stage() 

    prims = [stage.GetPrimAtPath(source_prim_path)]
    while len(prims) > 0:
        prim = prims.pop(0)
        if prim:
            if prim.GetTypeName() in ["Mesh", "Capsule", "Sphere", "Cube", "Cone", "Cylinder", "Disk", "Plane", "Torus"]:
                if not prim.IsInstance():
                    # prim.GetReferences().AddReference(assetPath=instance_usd_path, primPath=str(prim.GetPath()))
                    prim.SetInstanceable(True)
                # parent_prim = prim.GetParent()
                # if parent_prim and not parent_prim.IsInstance():
                #     parent_prim.GetReferences().AddReference(assetPath=instance_usd_path, primPath=str(parent_prim.GetPath()))
                #     parent_prim.SetInstanceable(True)
                #     continue

            children_prims = prim.GetChildren()
            prims = prims + children_prims

    if save_as_path is None:
        omni.usd.get_context().save_stage()
    else:
        omni.usd.get_context().save_as_stage(save_as_path)

convert_asset_instanceable(
    asset_usd_path="path_to_flattened_usd_file",
    source_prim_path="/source/prim",
    save_as_path="path_to_where_you_wanna_save_it"
)

[Yasu31]

Thanks!! if anyone wants to just apply the changes via GUI, I've found that this is enough (though I haven't checked for the performance yet)

1. run the converter script, which also opens the IsaacSim window python scripts/tools/convert_mjcf.py simple_gripper.xml simple_gripper.usd --fix-base --import-sites --make-instanceable
2. uncheck "Instanceable" for the Xform, and then check it for the geoms (the "I" indicates it's instanceable)

before	after

3. save it