Description
Environment
Version / Commit SHA: fvdb, 0.2.0
Describe the bug
Exception has occurred: RuntimeError
CUDA error: an illegal memory access was encountered
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
File "training/1gpu.py", line 280, in
scaler.scale(total_loss / grad_accum_steps).backward()
RuntimeError: CUDA error: an illegal memory access was encountered
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
To Reproduce
This is caused in the pytorch backward pass.
I'm not exactly sure which operation caused the problem, but I suspect it's related to sparse conv3d (i.e., fvnn.SparseConv3d).
When I remove this convolution, the issue doesn't occur anymore.
Of course, not all usages of convolution cause the problem — I suspect it happens when GPU memory usage is high, which triggers PyTorch's memory cache release.
It seems that SparseConv3d's internal cache isn't preserved correctly in this case, leading to an invalid memory access during the backward pass.