[Question] Correct way to load visual geometry (no physical properties) and corresponding collision geometry for massively parallel robot training

[jaraujo98]

Question

I want to train robots on a complex scene asset (many meshes, materials, and textures; several MB in size). I have reviewed the documentation and relevant code, and have tried several alternatives for days, but still don’t see a definite answer to the following:

---

1. Best way to load a large static visual-only scene for parallel training

What is the correct (or recommended) approach for loading a large, purely visual scene mesh (no physics, static geometry) so that it can be used efficiently across many parallel environments? Specifically:

• Can the scene be loaded once and shared by all environments (e.g., by setting env_spacing = 0), or would that lead to interference (collisions, showing up on the wrong render) between envs?
• Is converting the scene into an instanceable asset the only way to achieve this? If it must be instanceable:
  • Can the entire scene be made a single instance, or does each mesh/material need to be its own instanced asset?
  • If an instance is required per environment, do I need to manually apply the environment’s origin transform to each instance?

My scene is in GLB format link. When I use the mesh converter tool and load the resulting URDF half of the geometry disappears. If I convert it to OBJ the mesh converter sort of works, but I need to manually fix a broken path to get the scene to show up in the stage.

---

2. Best way to load custom collision geometry

I also created custom collision geometry for the scene using only box meshes (so all convex). These boxes are currently stored as meshes in a single file.

What is the recommended way to load these as collision geometry in Isaac Lab?

• How should I write the configuration for importing the mesh file directly?
• How do I know what sort of collision mesh processing will be applied (ex: convex decomposition, etc.) and how can I control it?
• Should I write a script to convert the boxes into a URDF with a single static link containing multiple <collision> elements? I tried something along those lines, but when I imported that URDF nothing showed up in the stage. Does there need to be a fixed joint between the world link and the link with the collision meshes to ensure the parser sees it?

---

3. Build Information

Due to hardware limitations I am currently using:

• Isaac Lab 2.1.1
• Isaac Sim 4.5

But answers for the latest versions are welcome too.

[RandomOakForest]

For training robots on a complex, static visual scene in Isaac Lab or Isaac Sim, loading and sharing large visual assets and custom collision geometry efficiently across parallel environments is a common challenge. I'll move this post to our Discussions for follow up. Here is a summary that may be of help.

Loading a Large Static Visual Scene Mesh (Visual-Only)

Best Practice:

• Load the visual scene once as a static asset to save memory and initialization time, rather than duplicating it for each parallel environment.
• Do not place the scene inside each environment’s root; instead, place it directly under /World so it’s outside environment transforms and not affected by per-env spacing.
• The visual scene should be marked as static (no physics) and as "visual-only". This means you do not create a simulation actor or rigid body, just a USD prim with meshes/materials.

Parallel Training Considerations:

• env_spacing = 0 works if each environment only contains dynamic objects and sensors. The static visual scene can be global, but each env must not have objects that collide or visually overlap in the same location as others.
• Sharing: Since static visual geometry is not part of the physics simulation, there’s no interference or collision across environments. Visibility issues only arise if dynamic objects from different envs overlap spatially (which they shouldn’t if envs are properly spaced or if only sensors interact with the static scene).

Instanceable Assets:

• Instanceable assets are only required if you want to duplicate the scene multiple times (e.g., for randomized backgrounds per env). For typical training, loading the scene once in /World and using per-env transforms for dynamic objects is preferred.
• You can make the entire scene a single instanceable asset (using USD’s instanceable prim feature), not per mesh/material—unless memory or render optimization needs dictate further instancing granularity.

Transform Handling:

• If you do choose to instance the entire scene per env, each instance must have its transform set to the respective env origin. However, for static global scenes, this is not needed—just import the scene at a fixed location and let environments manage their local sensors.
• For most RL and imitation learning setups, keeping the static scene at the origin and spacing environments for their dynamic content (robot + objects) is the cleanest solution.

GLB/OBJ Import Issues:

• The mesh converter tool can have quirks; always verify the output USD with the Isaac Sim Mesh Converter.
• Disappearing geometry can result from unsupported features (e.g., non-manifold meshes, unsupported materials). OBJ is generally more reliable for static scenes, but USD is preferred if you can directly export to it.
• After conversion, always check mesh paths and fix broken references in the resulting USD file. Blender USD export is recommended if you continue to see geometry issues.

Loading Custom Collision Geometry (Boxes as Meshes)

Recommended Workflow:

1. Primitive-based Colliders: Instead of importing meshes as colliders, define each box as a USD PhysicsCollisionAPI prim or as a primitive shape (GeomBox) if possible. This is more efficient and accurate for convex shapes.
2. Importing Meshes for Collision:
   • Place all collision boxes in one USD file and mark each mesh with PhysicsCollisionAPI.
   • Specify each box’s transform and ensure they are grouped under a single Xform or physics link if treating as one static object.
3. URDF Approach:
   • You can generate a URDF with a single link containing multiple <collision> elements, each referencing a box mesh. No joint is needed for a fixed scene; just a single link with all <collision> children.
   • If importing into Isaac Sim, ensure your URDF config uses static=True and does not assign mass/inertia or create physics actors if not needed.
   • If nothing appears on import, check for missing mesh file paths or scale issues. Visualize the imported USD in the Isaac Sim GUI for debugging.

Controlling Collision Mesh Processing:

• By default, Isaac Sim will treat box meshes as convex, unless non-convex decomposition is triggered via asset import options.
• Set import heuristics in your asset configuration file or the import dialog (e.g., disable convex decomposition if all shapes are already convex).
• Scripting: You can write a Python script to programmatically add collision boxes as USD prims with physics API attached; this gives full control and allows you to batch-generate static geometry.

Fixed Joints:

• For static scene geometry, you do not need a fixed joint from world unless you are importing a dynamic object with multiple rigid bodies. A single static link with all collision elements suffices.