on-the-fly conversion of URDF -> MuJoCo MJCF

[saikishor]

Spoiler alert 🚨

We are currently working on the MuJoCo MJCF generation on the fly. We had a brainstorming session with Ortisa today. The idea is to reuse your work and add some more args to convert URDF on the fly. One of the features, we discussed is to set the scene file at run time, so we can also include this part into the scene or add a key pose part into the MJCF directly.

Would that work for you guys?

I'm trying to avoid hardcoding too many things in the URDF, because that way we lack flexibility to start with different poses.

This would be awesome! I think we would be happy to share our thoughts about on the fly conversion if you are interested. We had thought about that, but decided against it for several reasons. It might make sense to talk about it in a different issue though.

Regardless, providing a scene file at run time would definitely be nice! We have also thought about this a bit, but our infrastructure usually has a separate ROS package for a specific application, and we haven't figured out how to properly combine assets that need to be in that package with the assets in a base robot package. But obviously creating stuff on the fly might help solve that!

Originally posted by @ndunkelb-nasa in #34 (comment)

[saikishor]

@ndunkelb-nasa @eholum-nasa please share your thoughts on on-the-fly conversion here. For us, we are interested in having it, because for the quite number of variants we have with our robots, it is easier for us to generate them on the fly from the maintenance perspective and a single source of information point of view as well.

[ndunkelb-nasa]

(Note these are from @eholum-nasa and myself)

We are definitely interested in doing this! However, we do have several concerns about trying to do this on-the-fly, and some more concerns just about some assumptions we have made. We would love to know if you have thought about some of these!

Some concerns about on-the-fly conversion

• Collision objects that are non-convex need to be decomposed into convex hulls. We use obj2mjcf, along with coacd arguments to do this. This often takes quite a long time to do successfully for large or complicated objects (up to maybe a minute or so), so would prefer to not run that every time.
• This will generate quite a lot of files to run, and might end up with quite a large output. Where do all of these files go? Just in the install space? Do they get overwritten every time? Maybe some of them can be cached if there are no changes (but that sounds like a ton of work).
• It is nice to see the final generated mjcf in a single file, to understand what all of the components did and troubleshoot issues. Without that, it would probably be really hard to understand when there are issues. Would definitely be good to support some kind of saving off the final file. I like to spend some time with the generated mjcf just running from the base mujoco simulate binary because I can hot-reload feature which helps me figure out spacing and other features before making it a fully automated process.
• Sometimes it is just unreasonable to assume we can get to a final version directly from the urdf. I have a post-processing python script that I run on one of my setups (adding in mecanum wheels geometry instead of default cylinders from urdf side), but it might be hard to layer on something like that in a standardized automated format.
  • Another example is if you want to apply some properties to a mesh (that doesn't have a name), you can't do that in the current setup
• We've kind of abused the idea of the "inputs file" for the converter, but how does one otherwise handle information that can't be easily parsed or included from an URDF? We thought about adding additional tags to URDFs that are mujoco driver specific but would prefer not to further abuse that standard. These are everything from sensors to friction parameters to actuator configuration and gains.
• Mujoco's converter collapses fixed bodies into a single element of the tree, so we lose information about sites that might be relevant from the URDF. Specifically things like force/torque sensor locations or camera frames. We handle those manually now by just re-adding every single link as a site in the final MJCF but maybe there is a more intelligent way to handle that.

Some general questions that could be improvements.

• CoACD is currently non-deterministic, so you get different meshes and colors on each output which is annoying. It would be nice to fix this. We already have an issue open for this.
• Should we be using collision or visual meshes?
  • We decided that the visual meshes are typically the higher fidelity option, so we defaulted to use them.
  • maybe it should be a choice per mesh?
• Should we be able to selectively leave some things as stl/obj?
  • We only convert things to obj so that we can run the decomposition on them
  • We also rely on blender's python bindings which aren't available on all platforms and are pretty heavy, we need to find an alternative
• Meshes come in without names as far as mujoco is concerned. We could name them according to the link that they came from so that we can reference them? Although that is based on the default urdf to mujoco converter which we don't really want to change

[saikishor]

@ndunkelb-nasa We will go through them and get back to you. Thank you