CUDA error “no kernel image is available for execution on the device” when using fused_sim from 3dgrut inside Docker

[viktorho]

Hi, thanks a lot for your team hardworks, beside, can you guys guide me to encounter this errors while build this application within a docker image?

Description

I’m encountering the following CUDA runtime error when running fused_sim from NVIDIA’s 3dgrut library inside a Docker environment:

RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Important note:
The exact same code works correctly in a Conda environment. The error appears only when running inside Docker.

---

Environment

• GPUs: 2× NVIDIA H100
• PyTorch: 2.1.2
• **3dgrut version: ** latest
• Cuda version: 11.8

---

What I Have Tried

• Checked the architecture list used by PyTorch:

  echo "TORCH_CUDA_ARCH_LIST=$TORCH_CUDA_ARCH_LIST"
  

  Output:

  7.0;7.5;8.0;8.6;9.0
  

  → SM 90 (Hopper) is included.

• Rebuilt the Docker image multiple times with different settings.

• Verified that outside Docker (Conda environment) everything runs fine → issue seems isolated to how CUDA kernels or extensions are compiled inside the container.

[rahul-goel]

Can you try specifying the architecture that you want to compile for using the environment variable CUDA_ARCHITECTURES as updated in the latest version of the repository?

Let me know if it doesn't work that way.

Thanks.