[Bug Report] Unitree Go2 training extremely slow on rough terrain

[yang-zj1026]

Describe the bug

When training RL locomotion policy of Unitree Go2 on rough terrains, the environment fps is significantly slow, around ~40k on 4090, while the environment fps of other robots like Unitree H1 is ~160k.

Steps to reproduce

./isaaclab.sh -p scripts/reinforcement_learning/rsl_rl/train.py --task Isaac-Velocity-Rough-Unitree-Go2-v0 --headless

System Info

Describe the characteristic of your environment:

• Commit: b5fa0eb
• Isaac Sim Version: 4.5.0
• OS: Ubuntu 22.04
• GPU: RTX 4090
• CUDA: 12.4
• GPU Driver: 565.57.01

Additional context

Add any other context about the problem here.

Checklist

• I have checked that there is no similar issue in the repo (required)
• I have checked that the issue is not in running Isaac Sim itself and is related to the repo

Acceptance Criteria

Add the criteria for which this task is considered done. If not known at issue creation time, you can add this once the issue is assigned.

• Criteria 1
• Criteria 2

[LZJ910]

I'm encountering the same issue: Isaac Sim 4.2 runs relatively slow, while Isaac Sim 4.5 performs even slower.

[RandomOakForest]

Thank you for posting this. Have you tried the recommended 535.129.03 driver? Alternatively, this doc with speed-up suggestions may be of help.

[Nate711]

I've also found that training on the rough terrain is incredibly slow, while flat terrain is fast.

With my 5090 and driver 570.124.06 I get 38,000 env steps/sec for rough and 235,000 for flat.

MJX gets 90k env steps/sec for a Go1 getup task which is somewhat similar (robot has many collision primitives) https://playground.mujoco.org/assets/playground_technical_report.pdf

[ubgk]

Thank you for posting this. Have you tried the recommended 535.129.03 driver? Alternatively, this doc with speed-up suggestions may be of help.

@RandomOakForest, I have the latest drivers installed, yet the problem persists.

I have fiddled a little more with IsaacLab, and realised that Go2 Rough terrain training takes ~%50 longer after updating the pip-installed isaacsim dependencies from v4.2.0.2 to v4.5.0.

Do you know what could be materially different between the two IsaacSim versions such that one is much slower?

My environment specs are as follows:

• Commit: 477b6a9
• Isaac Sim Version: 4.5.0
• OS: Ubuntu 22.04
• GPU: RTX 4070
• CUDA: 12.8
• GPU Driver: 570.124.04

[Nate711]

@ubgk Thanks for the extra test. Are you saying that updating isaacsim from 4.2 to 4.5 reduced the training speed by 50% or that just updating the dependencies from what's required in 4.2 to what's required in 4.5 caused that slowdown?

[ubgk]

@Nate711 sorry for the inaccurate wording. I actually meant the training takes 50% longer (i.e. 33% slower).

As for reproducing the issue, you may take a look at an earlier IsaacLab commit (sorry I lost the hash of such a commit, will try to look it up and edit this message) that's compatible both with IsaacSim 4.2.0.2 and 4.5.0 and test with both:

$ pip install isaacsim==4.2.0.2 isaacsim-extscache-physics==4.2.0.2 isaacsim-extscache-kit==4.2.0.2 isaacsim-extscache-kit-sdk==4.2.0.2 --extra-index-url https://pypi.nvidia.com

and

$ pip install isaacsim[all,extscache]==4.5.0 --extra-index-url https://pypi.nvidia.com

You will see that the latter is 33% slower on the Isaac-Velocity-Rough-Unitree-Go2-v0 environment. (With all IsaacLab remaining the same).

[RandomOakForest]

Thank you for following up. I will move this to our Discussions section. Let us know if we can be of further help.

[LZJ910]

I conducted rough terrain tests on the GO2 robot using both Isaac Sim 4.2 (orange) and 4.5 (blue). The attached results show performance comparisons between these two simulator versions.

[Nate711]

That's a huge difference! Thanks for measuring this. @RandomOakForest

[kellyguo11]

Hi all, thanks for the report and investigation! The performance difference in Isaac Sim 4.5 is caused by the new wheeled collision feature introduced in the physics simulation which will now allow for smoother contacts between cylinder and cone collision geometries with triangle meshes. This feature also comes at a performance cost, which is what we now see in the Go2 rough terrain environment as the robot has some cylinder collision meshes on the legs.

If you would like to go for faster performance without the improved contact behavior, you can add --kit_args="--/physics/collisionApproximateCylinders=true" to the command when running training, which should approximate the cylinders with convex meshes instead.

[Nate711]

Thanks @kellyguo11 for tracking down the issue. Is a faster cylinder collision routine on the roadmap? Mjx and mujoco both have fast cylinder contact resolution.

[Nemantor]

Is there a description of how the improved contact model processes contacts?