cuDNN issue

[mhmmd22]

i have build the darknet, darkhelp and darkmark on my ubuntu 24.0.4 ubuntu machine, i have installed the cuda tuoolkit, cuda drivere and cudnn library. i also installed LegoGears dataset to test the framework. I have a problem which is when i train the model i have created with the command darknet detector -map -dont_show train i get the next error

uDNN status error in /home/muhammed-nur/src/darknet/src-lib/dark_cuda.cpp, cudnn_handle(), line #167

---

• Errno 22: Invalid argument
• Error location: /home/muhammed-nur/src/darknet/src-lib/dark_cuda.cpp, cudnn_handle(), line #167
• Error message: cuDNN current error: status=2000, CUDNN_STATUS_BAD_PARAM
• Thread #127228683587584: main darknet thread
• Version "Slate" v4.0-52-g5150ae50 built on Aug 19 2025 17:26:33

---

i did not understand why i am getting this problem even if i have cuDNN, i have tested if my gpu works on tensorflow framework which uses python it detects my gpu without any problems

[stephanecharette]

Can you post the output of the following:

• darknet --version
• nvidia-smi
• nvcc --version

Have you tried rebooting? Same thing?

[mhmmd22]

Can you post the output of the following:

• darknet --version
• nvidia-smi
• nvcc --version

Have you tried rebooting? Same thing?

for darknet --version
Darknet V4 "Slate" v4.0-52-g5150ae50
Darknet is compiled to use the CPU. GPU is disabled.
OpenCV v4.6.0, Ubuntu 24.04
AVX detected.
FMA & AVX2 detected.

for nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Tue_Oct_29_23:50:19_PDT_2024
Cuda compilation tools, release 12.6, V12.6.85
Build cuda_12.6.r12.6/compiler.35059454_0

for nvidia-smi
Fri Aug 22 11:51:17 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.163.01 Driver Version: 550.163.01 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce GTX 1050 Off | 00000000:01:00.0 Off | N/A |
| N/A 48C P8 N/A / ERR! | 7MiB / 4096MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 2154 G /usr/lib/xorg/Xorg 4MiB |
+-----------------------------------------------------------------------------------------+

i tried to reboot the system but it did not work

[stephanecharette]

In regards to this:

for darknet --version
Darknet V4 "Slate" v4.0-52-g5150ae50
Darknet is compiled to use the CPU. GPU is disabled.

Note you did not have CUDA and CUDNN available when you built Darknet. It is only compiled to run on your CPU. You need to follow the instructions in the readme and rebuild Darknet. Remember to delete your cmake cache file to force cmake to re-check for CUDA and CUDNN.

[mhmmd22]

In regards to this:

for darknet --version
Darknet V4 "Slate" v4.0-52-g5150ae50
Darknet is compiled to use the CPU. GPU is disabled.

Note you did not have CUDA and CUDNN available when you built Darknet. It is only compiled to run on your CPU. You need to follow the instructions in the readme and rebuild Darknet. Remember to delete your cmake cache file to force cmake to re-check for CUDA and CUDNN.

tahnks for your feedback i will do this and try again

[mhmmd22]

In regards to this:

for darknet --version
Darknet V4 "Slate" v4.0-52-g5150ae50
Darknet is compiled to use the CPU. GPU is disabled.

Note you did not have CUDA and CUDNN available when you built Darknet. It is only compiled to run on your CPU. You need to follow the instructions in the readme and rebuild Darknet. Remember to delete your cmake cache file to force cmake to re-check for CUDA and CUDNN.

i still have the probelm whihc i described first. i searched about cuda and i saw that my gpu compute capability is 6.1 from this website https://developer.nvidia.com/cuda-legacy-gpus i thought that my gpu does not work on new versions of cuda toolkit so i installed cuda toolkit 11.8, after changing the version of cuda tookit i reinstalled the darknet framework and i got this messege for cmake -DCMAKE_BUILD_TYPE=Release ..

-- Darknet v4.0-52-g5150ae50
-- Darknet branch name: master
-- The C compiler identification is GNU 13.3.0
-- The CXX compiler identification is GNU 13.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for a CUDA compiler
-- Looking for a CUDA compiler - NOTFOUND
CMake Warning at CM_dependencies.cmake:52 (MESSAGE):
Support for NVIDIA CUDA not found.
Call Stack (most recent call first):
CMakeLists.txt:34 (INCLUDE)

-- Looking for a HIP compiler
-- Looking for a HIP compiler - NOTFOUND
CMake Warning at CM_dependencies.cmake:135 (MESSAGE):
Support for AMD/ROCm/HIP not found.
Call Stack (most recent call first):
CMakeLists.txt:34 (INCLUDE)

CMake Warning at CM_dependencies.cmake:146 (MESSAGE):
Neither NVIDIA CUDA nor AMD ROCm detected. Darknet will be CPU-only.
Call Stack (most recent call first):
CMakeLists.txt:34 (INCLUDE)

-- Hardware is 32-bit or 64-bit, and seems to be Intel or AMD: x86_64
-- Compiler: GNU/Clang=1 GNU/Clang/MSVC=TRUE: GNU
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Found Threads
-- Found OpenCV: /usr (found version "4.6.0")
-- Found OpenCV 4.6.0
-- Found OpenMP
-- Enabling AVX and SSE optimizations.
-- Making an optimized release build.
-- Skipping Doxygen (not found)
-- Setting up DARKNET OBJ
-- Setting up DARKNET LIB
-- Setting up DARKNET CLI
-- Setting up DARKNET examples
-- Configuring done (1.0s)
-- Generating done (0.0s)
-- Build files have been written to: /home/muhammed-nur/src/darknet/build

is that normal for only changing the version of cuda toolkit ?

[stephanecharette]

-- Looking for a CUDA compiler
-- Looking for a CUDA compiler - NOTFOUND

Normally, the cause of this is that nvcc is not installed or not on the PATH. You are certain that in the same shell you are running cmake you can also run nvcc --version and nvidia-smi without getting an error? Same user, same shell?

[mhmmd22]

-- Looking for a CUDA compiler
-- Looking for a CUDA compiler - NOTFOUND

Normally, the cause of this is that nvcc is not installed or not on the PATH. You are certain that in the same shell you are running cmake you can also run nvcc --version and nvidia-smi without getting an error? Same user, same shell?

İ think i didn't explain very well. İ solved the problem of cuda is not found by replacing cuda toolkit with newer version, and darknet detected so, darknet did not show this messege and can recognize my gpu

The problem i mentioned is this
uDNN status error in /home/muhammed-nur/src/darknet/src-lib/dark_cuda.cpp, cudnn_handle(), line #167

Errno 22: Invalid argument

Error location: /home/muhammed-nur/src/darknet/src-lib/dark_cuda.cpp, cudnn_handle(), line #167

Error message: cuDNN current error: status=2000, CUDNN_STATUS_BAD_PARAM

Thread #127228683587584: main darknet thread

Version "Slate" v4.0-52-g5150ae50 built on Aug 19 2025 17:26:33

This happens when i try to train my model. İ didn't understand why this happened

İ am using gtx1050 ti gpu, i think this may gives some insights

[stephanecharette]

Please post the output of darknet --version.

Then also answer these questions:

• the width and height of your neural network
• the base configuration you are using
• the number of subdivisions you are using
• the iteration where the error happens

[mhmmd22]

Please post the output of darknet --version.

Then also answer these questions:

• the width and height of your neural network
• the base configuration you are using
• the number of subdivisions you are using
• the iteration where the error happens

for darknet --version
Darknet V4 "Slate" v4.0-52-g5150ae50
CUDA runtime version 12090 (v12.9), driver version 12090 (v12.9)
cuDNN version 13000 (v9.12.0), use of half-size floats is ENABLED
=> 0: NVIDIA GeForce GTX 1050 [#6.1], 3.9 GiB
OpenCV v4.5.4, Ubuntu 22.04
width: 352
height: 256
configuration: yolov4_tiny.cfg
subdivision: 4

the error happens before the training starts

[stephanecharette]

Thanks for the information.

the error happens before the training starts

Can you add -trace to the flags when you train? For example:

darknet detector -trace -map -dontshow train LegoGears.data LegoGears.cfg > output.txt

Please attach the output.txt file so I can get a better idea of what is happening.

[mhmmd22]

Thanks for the information.

the error happens before the training starts

Can you add -trace to the flags when you train? For example:

darknet detector -trace -map -dontshow train LegoGears.data LegoGears.cfg > output.txt

Please attach the output.txt file so I can get a better idea of what is happening.

i attached the output file

output.txt

[stephanecharette]

Normally, the CUDNN_STATUS_BAD_PARAM error is due to out-of-vram problems, or algorithm selection. This normally means you don't have enough vram on your GPU. But it normally doesn't present exactly the way it is in the output you've attached.

I would try and increase your subdivisions some more, or try on a system with a different GPU with more vram. For example, with the output you've provided I see you're using 4 subdivisions on line #18. Try and increase this to 8 or 16 for example to see if the error remains the same.

[mhmmd22]

Normally, the CUDNN_STATUS_BAD_PARAM error is due to out-of-vram problems, or algorithm selection. This normally means you don't have enough vram on your GPU. But it normally doesn't present exactly the way it is in the output you've attached.

I would try and increase your subdivisions some more, or try on a system with a different GPU with more vram. For example, with the output you've provided I see you're using 4 subdivisions on line #18. Try and increase this to 8 or 16 for example to see if the error remains the same.

I tried increasing the subdivision size, but it didn’t work. I also changed the dimensions of my network, but nothing changed.