Are flexible objects supported now in Unity Plugin?

[AnnikaFAU]

Intro

Hi!

I am a master's student at FAU Erlangen-Nürnberg, I use MuJoCo for my research on deformable object dynamics in combination with VR.

My setup

• Unity 6000.3.9f1
• MuJoCo 3.6.0
• Unity Plugin: 3.6.0
• OS: Windows 11

My question

I am attempting to import .xml files from the mujoco-3.6.0/model/flex directory (e.g. /softbox.xml) using Assets > Import MuJoCo Scene in Unity. Since the flex plugins were merged into version 3.6.0 via a pull request (or even earlier), I expected these models to load and function correctly. However, I am instead encountering the attached error messages.

I have thoroughly searched for existing discussions on this issue, but the available resources appear to be outdated or refer to significantly older versions (e.g. #2008 ).

I would appreciate clarification on the following points:

• The error messages indicate that flex is not yet supported. Could you confirm whether this is indeed the case, or if there might be an issue with my configuration? I am particularly confused by the presence of elasticity.dll if the Unity importer cannot process flex .xml files.

• If flex is not currently supported, are there alternative approaches for implementing deformable objects in Unity using MuJoCo?

• Has anyone successfully combined the MuJoCo plugin with XR Interaction (e.g., Meta Quest 3) in Unity? Specifically, is it possible to interact with MuJoCo objects in this setup?

Thank you very much for your time and assistance. I look forward to your response.

Imported MJCF loaded, saved to C:/Users/XXX~1/AppData/Local/Temp/DefaultCompany/MuJoCo\softbox863.xml
UnityEngine.Debug:Log (object)
Mujoco.MjImporterWithAssets:ImportFile (string) (at ./Packages/unity/Editor/Importer/MjImporterWithAssets.cs:56)
Mujoco.MjImporterEditorWindow:Apply () (at ./Packages/unity/Editor/Importer/MjImporterEditorWindow.cs:28)

Meshes locations: source = C:\Users\XXX\MuJoCo3.6\mujoco-3.6.0\mujoco-3.6.0\model\flex\asset/, target = C:/Users/XXX/MuJoCo3.6/Assets\Local\MjImports\softbox863\Resources
UnityEngine.Debug:Log (object)
Mujoco.MjImporterWithAssets:ConfigureMeshPath (string,string,System.Xml.XmlDocument) (at ./Packages/unity/Editor/Importer/MjImporterWithAssets.cs:119)
Mujoco.MjImporterWithAssets:ImportString (string,string,string) (at ./Packages/unity/Editor/Importer/MjImporterWithAssets.cs:72)
Mujoco.MjImporterWithAssets:ImportFile (string) (at ./Packages/unity/Editor/Importer/MjImporterWithAssets.cs:60)
Mujoco.MjImporterEditorWindow:Apply () (at ./Packages/unity/Editor/Importer/MjImporterEditorWindow.cs:28)

The importer does not yet support tags .
UnityEngine.Debug:Log (object)
Mujoco.MjcfImporter:ParseBodyChild (System.Xml.XmlElement,UnityEngine.GameObject) (at ./Packages/unity/Runtime/Importer/MjcfImporter.cs:293)
Mujoco.MjcfImporter:ParseBodyChildren (UnityEngine.GameObject,System.Xml.XmlElement) (at ./Packages/unity/Runtime/Importer/MjcfImporter.cs:237)
Mujoco.MjcfImporter:ParseRoot (UnityEngine.GameObject,System.Xml.XmlElement) (at ./Packages/unity/Runtime/Importer/MjcfImporter.cs:147)
Mujoco.MjImporterWithAssets:ParseRoot (UnityEngine.GameObject,System.Xml.XmlElement) (at ./Packages/unity/Editor/Importer/MjImporterWithAssets.cs:91)
Mujoco.MjcfImporter:ImportXml (System.Xml.XmlDocument,string) (at ./Packages/unity/Runtime/Importer/MjcfImporter.cs:100)
Mujoco.MjImporterWithAssets:ImportString (string,string,string) (at ./Packages/unity/Editor/Importer/MjImporterWithAssets.cs:75)
Mujoco.MjImporterWithAssets:ImportFile (string) (at ./Packages/unity/Editor/Importer/MjImporterWithAssets.cs:60)
Mujoco.MjImporterEditorWindow:Apply () (at ./Packages/unity/Editor/Importer/MjImporterEditorWindow.cs:28)

The importer does not yet support tags .
UnityEngine.Debug:Log (object)
Mujoco.MjcfImporter:ParseBodyChild (System.Xml.XmlElement,UnityEngine.GameObject) (at ./Packages/unity/Runtime/Importer/MjcfImporter.cs:293)
Mujoco.MjcfImporter:ParseBodyChildren (UnityEngine.GameObject,System.Xml.XmlElement) (at ./Packages/unity/Runtime/Importer/MjcfImporter.cs:237)
Mujoco.MjcfImporter:ParseRoot (UnityEngine.GameObject,System.Xml.XmlElement) (at ./Packages/unity/Runtime/Importer/MjcfImporter.cs:147)
Mujoco.MjImporterWithAssets:ParseRoot (UnityEngine.GameObject,System.Xml.XmlElement) (at ./Packages/unity/Editor/Importer/MjImporterWithAssets.cs:91)
Mujoco.MjcfImporter:ImportXml (System.Xml.XmlDocument,string) (at ./Packages/unity/Runtime/Importer/MjcfImporter.cs:100)
Mujoco.MjImporterWithAssets:ImportString (string,string,string) (at ./Packages/unity/Editor/Importer/MjImporterWithAssets.cs:75)
Mujoco.MjImporterWithAssets:ImportFile (string) (at ./Packages/unity/Editor/Importer/MjImporterWithAssets.cs:60)
Mujoco.MjImporterEditorWindow:Apply () (at ./Packages/unity/Editor/Importer/MjImporterEditorWindow.cs:28)

The importer does not yet support equality .
UnityEngine.Debug:Log (object)
Mujoco.MjcfImporter:ParseEqualityType (System.Xml.XmlElement) (at ./Packages/unity/Runtime/Importer/MjcfImporter.cs:315)
Mujoco.MjcfImporter:ParseRoot (UnityEngine.GameObject,System.Xml.XmlElement) (at ./Packages/unity/Runtime/Importer/MjcfImporter.cs:186)
Mujoco.MjImporterWithAssets:ParseRoot (UnityEngine.GameObject,System.Xml.XmlElement) (at ./Packages/unity/Editor/Importer/MjImporterWithAssets.cs:91)
Mujoco.MjcfImporter:ImportXml (System.Xml.XmlDocument,string) (at ./Packages/unity/Runtime/Importer/MjcfImporter.cs:100)
Mujoco.MjImporterWithAssets:ImportString (string,string,string) (at ./Packages/unity/Editor/Importer/MjImporterWithAssets.cs:75)
Mujoco.MjImporterWithAssets:ImportFile (string) (at ./Packages/unity/Editor/Importer/MjImporterWithAssets.cs:60)
Mujoco.MjImporterEditorWindow:Apply () (at ./Packages/unity/Editor/Importer/MjImporterEditorWindow.cs:28)

NullReferenceException: Object reference not set to an instance of an object
Mujoco.MjcfImporter.CreateGameObjectWithUniqueName (UnityEngine.GameObject parentObject, System.Xml.XmlElement parentNode, System.Type componentType) (at ./Packages/unity/Runtime/Importer/MjcfImporter.cs:123)
Mujoco.MjcfImporter.ParseRoot (UnityEngine.GameObject rootObject, System.Xml.XmlElement mujocoNode) (at ./Packages/unity/Runtime/Importer/MjcfImporter.cs:188)
Mujoco.MjImporterWithAssets.ParseRoot (UnityEngine.GameObject parentObject, System.Xml.XmlElement parentNode) (at ./Packages/unity/Editor/Importer/MjImporterWithAssets.cs:91)
Mujoco.MjcfImporter.ImportXml (System.Xml.XmlDocument mjcfXml, System.String name) (at ./Packages/unity/Runtime/Importer/MjcfImporter.cs:100)
UnityEngine.Debug:LogException(Exception)
Mujoco.MjcfImporter:ImportXml(XmlDocument, String) (at ./Packages/unity/Runtime/Importer/MjcfImporter.cs:104)
Mujoco.MjImporterWithAssets:ImportString(String, String, String) (at ./Packages/unity/Editor/Importer/MjImporterWithAssets.cs:75)
Mujoco.MjImporterWithAssets:ImportFile(String) (at ./Packages/unity/Editor/Importer/MjImporterWithAssets.cs:60)
Mujoco.MjImporterEditorWindow:Apply() (at ./Packages/unity/Editor/Importer/MjImporterEditorWindow.cs:28)

Minimal model and/or code that explain my question

Confirmations

• I searched the latest documentation thoroughly before posting.
• I searched previous Issues and Discussions, I am certain this has not been raised before.

[Balint-H]

Hello! Glad you are interested in deformable objects in Unity! I made the PR for the preliminary support. Flexes were supported as far as the example scenes went that were shipped with Mujoco at that time point (if I remember correctly, it has been a while #2325). Rendering of the flex meshes was incomplete, I think I had problems with the normals, but I had some support for it! There are some instructions on how to try it.

It was experimental, however it might work for your needs, or you could get it working with a bit of development! Your best option is to check out the PR version of the plugin (look into how to add a specific branch of a git package to your Unity project, but I can help if you get stuck on that). Importantly, add the mujoco and plugins that correspond to the same version (I think it was 3.2.5)!

Let me know if you have issues, I might be able to troubleshoot. However, I won't be able to active development on it until end of May (feel free to ping me here then!)

As far as VR/XR goes It is definitely possible!. With oculus/meta link it is quite straightforward, no different than doing a VR scene in Unity + doing a MuJoCo Scene. To interact physically, track the end effectors with MjMocapBodies, and use MjWeld constraints on collider bodies to track the MocapBodies. For standalone builds, you can use the android builds from @joanllobera, see #3195

One last thing, as far as alternate options instead of flex bodies, you can try composite bodies.

[AnnikaFAU]

Thank you for the fast and detailed answer. I tried out the feature/unity-plugins package together with the 3.2.5 dlls. The errors that occur when importing a flex .xml are solved. I also had to edit the MjMouseSpring.cs script. Now I have an error referring to the stability of the model.

• I get the following error, when running e.g. the model floppy.xml and others from the flex folder.

  PhysicsRuntimeException: BADQACC: NaN/inf in qacc.
  Mujoco.MjScene.CheckForPhysicsException () (at ./mujoco-feature-unity-plugins/mujoco-feature-unity-plugins/unity/Runtime/Components/MjScene.cs:380)
  Mujoco.MjScene.StepScene () (at ./mujoco-feature-unity-plugins/mujoco-feature-unity-plugins/unity/Runtime/Components/MjScene.cs:339)
  Mujoco.MjScene.FixedUpdate () (at ./mujoco-feature-unity-plugins/mujoco-feature-unity-plugins/unity/Runtime/Components/MjScene.cs:98)
  

• The material was also not available (pink). I changed it to URP.Lit.

• The CheckForPhysicsException() is failing at the following if-statement:

  if (Data->warning6.number > 0) {
  Data->warning6.number = 0;
  throw new PhysicsRuntimeException("BADQACC: NaN/inf in qacc.");
  }

I already tried changing the solver from CG to implicitfast and the timestep from 0.001 to 0.0005. I still get the error.

Do you have experience with this issue or any idea how to solve this?

[Balint-H]

Thanks for persevering!

The material issue was due to Unity deprecating their built in renderer, support for the new materials is now added in the newer versions of the plugin. Of course, that is not helpful for you as for the time being you should use this older version. Your fix of manually reassigning materials is indeed the correct one. Similarly, the error from the mouse spring was also from API changes in unity and are fixed in the newer versions, but good job on fixing that.

The number one cause of instabilities is the fixedTimestep. Check your project settings and make sure the physics timestep is sufficiently low (e.g. 0.001). If that resolves it, explore what is the highest timestep you can use that avoids instability.

[AnnikaFAU]

Thanks, that helped!

• I also added your script (MjFlexMeshBuilder.cs) and meshes to the objects. I tried it with the scene press.xml. The scene has a high computational load which leads to strong delays, when watching it via the VR headset (streaming function). Unfortunately this circumstance makes it almost useless for me.

• I also added interaction via the controllers. Therefore I added rigidbodies, boxcolliders, XR Grab Interactable and XR Grab Transformer to the press boxes. My plan was to take one box and deform a soft cube with it. Grabbing the press box worked. But even though the two objects collide in the scene during the pressing process, they won't collide, when I grab the press box and touch the soft cube with it. This might be because processing takes a lot of time.

• I can also send videos if you are interested

Please let me know if you have any idea, how to make them collide. And if you have an idea how to decrease the high delays in the VR.
Thank you :)

[Balint-H]

You can definitely grab mocapbodies with the XR tool! Thats how I have done it in my example video. The grabbed object can move around and push around MuJoCo simulated bodies.

MocapBodies are really just regular Unity objects that have the MjMocapBody component - but you can attach other scripts to them as well. The second method that you mentioned is exactly the type of way you could do this.

Could you test non-elasticity MuJoCo scene for comparison? E.g. model/humanoid/humanoid.xml or model/humanoid/humanoid100.xml?

These scenes ran at a good framerate when I did my VR tests.

[AnnikaFAU]

Yes, you are right. I tried it quick. XR Interaction works with the humanoid. And without delay.

[Balint-H]

Even humanoid100?

[AnnikaFAU]

Now it also works in the Jelly scene. I think I configured it wrong before. Thanks for correcting me.
Now there's only the delay problem left. It looks really choppy in the scene, rather like a flipbook that only has every tenth page in it and you flip really slow. 😄
And I always see this hourglass waiting symbol.

Do you think this problem is due to limited computer performance?
Would you say, that it can be solved/improved by choosing a more performant pc?

[Balint-H]

Absolutely, that can have a big effect. One last test I'd suggest you do is loading the humanoid and the jelly scenes using the Simulate tool mujoco comes bundled with (check the docs if you have not used it before)

If you load the xmls (e.g. by drag and dropping them onto the simulate app's opened window), how do they run? If they are choppy/slower than real time even there then the answers is definitely a hardware upgrade.

Of course in Unity you have some extra overheads with VR, the unity update loop and the state syncing, but this is I suspect not the main culprit here