Describe the bug
On the Newton backend, the first step() after a hard reset (NewtonManager.reset(soft=False)) can crash with a CUDA error 700 (illegal memory access).
Root cause: a hard reset re-finalizes the Newton Model, which reallocates its device arrays, but the existing CollisionPipeline is reused. The pipeline still holds references to the old model (via its cached self.model). Once the old model's device buffers are freed and reused — which happens under GPU memory pressure between resets in real workloads — the pipeline's next collide() dereferences freed/mismatched device memory and faults with CUDA 700.
The failure typically surfaces in the collision kernels compute_shape_aabbs or narrow_phase_kernel_gjk_mpr, on the first step() after the reset, entirely outside any CUDA graph.
Secondary impact: because this crash appears with CUDA graph capture enabled, a common workaround has been to disable CUDA graphs, which significantly degrades stepping performance. Fixing the root cause allows CUDA graph capture to stay enabled across hard resets.
Steps to reproduce
- Build a Newton collision scene (e.g. several free-falling boxes over a ground plane) using the Newton collision pipeline.
- Drive
reset -> step -> hard reset.
- Between resets, cause the old model's collision device buffers to be freed and the memory reused (real workloads do this via GPU memory pressure between resets).
- The next
step() (followed by a device sync) faults with CUDA error 700.
Error
CUDA error 700: an illegal memory access was encountered
(surfacing in compute_shape_aabbs / narrow_phase_kernel_gjk_mpr on the first step() after a hard reset)
Expected behavior
A hard reset should rebuild the collision pipeline against the re-finalized model so the first subsequent step() runs without any CUDA error, with CUDA graph capture either enabled or disabled.
System Info
- Backend: Newton (
isaaclab_newton)
- Solver: MJWarp (
MJWarpSolverCfg, Newton collision pipeline)
- GPU: NVIDIA L40 (also reproducible on a single GPU)
Additional context
The bug is independent of GPU count (it reproduces on a single GPU) and is not related to deformables — it is specific to the Newton rigid-body collision pipeline not being rebuilt on a hard reset.
Describe the bug
On the Newton backend, the first
step()after a hard reset (NewtonManager.reset(soft=False)) can crash with a CUDA error 700 (illegal memory access).Root cause: a hard reset re-finalizes the Newton
Model, which reallocates its device arrays, but the existingCollisionPipelineis reused. The pipeline still holds references to the old model (via its cachedself.model). Once the old model's device buffers are freed and reused — which happens under GPU memory pressure between resets in real workloads — the pipeline's nextcollide()dereferences freed/mismatched device memory and faults with CUDA 700.The failure typically surfaces in the collision kernels
compute_shape_aabbsornarrow_phase_kernel_gjk_mpr, on the firststep()after the reset, entirely outside any CUDA graph.Secondary impact: because this crash appears with CUDA graph capture enabled, a common workaround has been to disable CUDA graphs, which significantly degrades stepping performance. Fixing the root cause allows CUDA graph capture to stay enabled across hard resets.
Steps to reproduce
reset -> step -> hard reset.step()(followed by a device sync) faults with CUDA error 700.Error
Expected behavior
A hard reset should rebuild the collision pipeline against the re-finalized model so the first subsequent
step()runs without any CUDA error, with CUDA graph capture either enabled or disabled.System Info
isaaclab_newton)MJWarpSolverCfg, Newton collision pipeline)Additional context
The bug is independent of GPU count (it reproduces on a single GPU) and is not related to deformables — it is specific to the Newton rigid-body collision pipeline not being rebuilt on a hard reset.