Future plans for MuJoCo Warp renderer fidelity and camera sensor outputs?

[tkelestemur]

Hi MuJoCo Warp team,

We are evaluating MuJoCo Warp for vision-based RL / sim-to-real workflows where high-throughput batch rendering across many parallel environments is very appealing.

From the current renderer docs/API, my understanding is that the MuJoCo Warp batch renderer supports mesh rendering, textures, heightfields, flex rendering, heterogeneous multi-camera rendering, lighting/shadows, and outputs such as RGB, depth, and MuJoCo-style segmentation.

I’m wondering what the intended roadmap is for the renderer. Should users think of it primarily as a fast geometric/visual observation renderer for training, or is there an ambition to move toward Isaac Lab / Isaac Sim RTX-style camera simulation and synthetic-data generation?

In particular, compared with Isaac Lab / Isaac Sim cameras, the missing or unclear areas seem to be:

• Photorealistic / physically based rendering: PBR materials, specular/roughness/metallic properties, reflections/refractions, transparency, global illumination / ambient occlusion, softer shadows, HDR/tone mapping.
• Camera optics and sensor effects: lens distortion / fisheye / calibrated camera models, exposure/gamma/white balance, sensor noise, depth of field, rolling shutter, motion blur.
• Additional camera products / annotators: RGBA, distance-to-camera vs distance-to-image-plane depth, surface normals, motion vectors / optical flow, semantic segmentation, instance segmentation, 2D/3D bounding boxes, point clouds, and label metadata.
• Asset/material compatibility: richer texture/material support, normal maps, emissive materials, and broader compatibility with synthetic-data asset pipelines.
• Could you share whether any of these capabilities are planned, explicitly out of scope, or good candidates for community contribution?

My main use case is batched vision-based robot learning where I’d like to keep MuJoCo Warp’s throughput, but also understand how close the renderer is expected to get to photorealistic camera pipelines such as Isaac Sim / Isaac Lab RTX cameras. If the intended design is to stay lightweight and fast rather than photo-realistic, that would also be useful to know.

[yuvaltassa]

cc @StafaH

[StafaH]

Hi @tkelestemur,

Our goal when we originally built the renderer was to create a high throughput but lower fidelity renderer. In our experiments we are able to do sim2real with the lower fidelity rendering as well, although this is definitely an open research question. The strength of this renderer is that you can train vision-based policies fast and iterate quickly.

We don't have a roadmap for what's next, but we're happy to listen to feedback from what the community needs and build in that direction. One of the beautiful things about the renderer is that you can easily extend it, just build your own render kernel based off what you see in MJWarp. We've seen the newton folks do this and add cool new features like gaussian splat rendering with the same kernel design.

External contributions that improve the performance or add new features would definitely be appreciated, and can be discussed in advance. Some things might be beneficial to the main renderer, but specific features and renderer changes might be better living in your own project. As long as you have the same function signature as the MJWarp render kernel, you can use your own render function to fill the output buffers.