[QST] cudaErrorInsufficientDriver

[yangjianfengo1]

When I run examples/python/CuTeDSL/blackwell/dense_gemm.py on the Blackwell, I find the error CUDAError InsufficientDriver. My CUDA version is 12.9.41 and my CUDA driver is 570.133.20. May I ask if I need to upgrade the driver or CUDA version.

[brandon-yujie-sun]

@yangjianfengo1 could you try upgrading the driver per the CUDA 12.9 GA row in the table here: https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#id8

[jayhshah]

With version 4.3.0 and higher, I get this error for CUDA driver version 575.57.08. However, version 4.3.0.dev0 works. Everything works with CUDA driver version 580.95.05.

cutlass.base_dsl.common.DSLCudaRuntimeError: DSLCudaRuntimeError: cudaErrorInsufficientDriver (error code: 35) 

In addition, running through the docker container nvcr.io/nvidia/pytorch:25.05-py3 rather than a python venv, but still with 575.57.08 also works for some reason, but I don't know why.

All tests were done with a simple hello world kernel.

[jayhshah]

Minimal repro:

uv venv
uv pip install nvidia-cutlass-dsl
uv run hello_world.py

produces

root@To:~/repos/zzz# uv run hello_world.py 
CUDA Runtime Version: 12090
CUDA Driver Version: 12090
Running hello_world()...
Traceback (most recent call last):
  File "/root/repos/zzz/hello_world.py", line 43, in <module>
    hello_world()
  File "/root/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/functools.py", line 998, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "/root/repos/zzz/.venv/lib/python3.12/site-packages/nvidia_cutlass_dsl/python_packages/cutlass/base_dsl/runtime/cuda.py", line 292, in checkCudaErrors
    raise DSLCudaRuntimeError(error_code, error_name)
cutlass.base_dsl.common.DSLCudaRuntimeError: DSLCudaRuntimeError: cudaErrorInsufficientDriver (error code: 35) 

hello_world.py

[brandon-yujie-sun]

@jayhshah could you please check the version of cuda-python module you installed? For the failure you observed with 575 driver, if you have a 13.0+ cuda-python installed, it won't work as 13.0+ cuda-python requires 580+ driver. You would need to install a 12.9 cuda-python to make it work.

Everything works with CUDA driver version 580.95.05.

Yes this is expected like you observed. Either version of cuda-python would just work with a 580+ driver.

[jayhshah]

That was the issue, thanks for explaining it! It was cuda-python version 13.1.1. After downgrading to 12.9.4, it worked.

The strange behavior is then that it ran on nvidia-cutlass-dsl version 4.2.1 in the first place with cuda-python 13.1.1.