having an issue with my Nvidia GPU not being detected and it using the cpu instead

[pochrist]

OS Ubuntu 24.04 LTS fresh install
System:
Ryzen 5 4600G
RTX 3060 12GB
Asrock B550 Riptide
16GB Memory
1 TB NVME SSD

After the OS install I ran the obligatory sudo apt Update and Upgrade command
Ran the command to install nomad from github - after this the fun began
got some error about 'curl' not being a recognized - so i installed that
then GPU not being detected showed up, at this point enlisted Gemini Pro.... and after that it was a hour of - Try this.... nope didn't work, now lets try this....nope didn't work, a lot of back and forth try something, paste the error- rinse repeat.

At this point I am considering either and older version of Ubuntu or debian

Anybody else hit these snags?

[pochrist]

I installed the 22.04 version of ubuntu and ran into the same is with the Gpu being detected and but not beig check off in the list, however I installed a small AI model and used the watch -n 1 nvidia-smi in terminal and started a conversation with the AI and watched the gpu resources shoot up to the 90's so it is obviously working... I guess the installer has some bugs still. Hope this helps other seeing the same issue.

[PurpleBanana-ai]

I have dealt with Ubuntu/Debian driver issues for the last few years, I would be more than happy to help or at least offer suggestions. One of the issues I had when I installed Nomad on my secondary AI machine, when the install script ran the docker portion forced an upgrade of the drivers. This caused my cards to not show up at all anymore under nvidia-smi because it was not in tune with kernel.
The problem was two fold:

The drivers installed were not the ones that supported on my systems setup, which requires Cuda Compute 8.6 and 12.0 (RTX 3090 & RTX 5080). For your 3060 you have to make sure 8.6 is supported by the driver (570 or 580, proprietary version). 590 may work. Also, verify the version of Cuda installed under nvidia-smi, is it 12.x or 13.x?

The kernel out of whack was the second problem after the install script installed the new Cuda drivers. This required updating the headers and then rebuilding the kernel (this also worked on Debian 13 for my main AI PC for dealing with a mixture of blackwell and ampere architecture cards).

I AM NOT AN EXPERT, PLEASE RESEARCH THIS ONE BEFORE YOU TRY IT OR MAKE SURE YOU KNOW HOW TO GET BACK TO A PREVIOUS KERNEL IN THE GRUB MENU! Just my caveat and disclaimer lol, if it is one thing I have learned is that nothing in life or tech "just works" like people say, especially when its free ;-).

In the Terminal try these -

These just look for data, they don't change anything:

1. First check the header version: uname -r
2. Then: ls /lib/modules/
3. Now run: sudo dmesg | grep -i nvidia | head -20
4. Lastly: lsmod | grep nvidia

What you are looking for is to see if you booted into a new kernel but the new nvidia modules didn't load. Compare the version of the kernel from 1. above to what is listed in number 2 to see if they are different.
Then, what you want to see is number 3 showing you a bunch of nvidia listings like "[2.213099] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:1c.4/0000:08:00.1/sound/card2/input13".
Now, if number 4 doesn't return any values, what happened was your kernel was rebuilt but without the correct headers, the cards are still detected though based on the output of number 3.

Then try 2, this will attempt to modify and install but will most likely error (this is where most of the AI's end up going off base if it errors):

1. sudo dkms status
2. sudo dkms autoinstall
   Output of 1 for me was: nvidia/595.45.04, 6.12.73+deb13-amd64, x86_64
   Output of 2 errored: Note (this is from my Debian one, Ubuntu was the same issue but I did have to verify AUTOINSTALL=yes in the dkms.conf file).

THE FIX FOR ME
My Fix on Ubuntu 24 and Debian 13 was to run these. THESE DO MODIFY:

1. sudo apt install linux-headers-$(uname -r)
2. sudo dkms autoinstall
3. If dkms succeeds then: sudo modprobe nvidia
4. nvidia-smi
   My cards then were picked up again although I was now running on driver 595 and Cuda 13.2, which is fine for me and my software stack which contains things like: vLLM, llama.cpp, Ollama, ComfyUI, Unsloth/Huggingface (fine-tuning), mem0, Docling and Langchain (RAG pipelines), OpenWebUI, etc., basically everything requiring vector db's, pytorch and cuda dependent, plus all of my hardware builds are "frankenbuilds" so I have five cards in one of them (two external via m.2 occulink). Thankfully I stocked up before the great "RAM/VRAM" ripoff going on now, no way affordable for the majority now.

I am not a fan of docker and rarely use it but I understand why it is used for a lot of rollouts and end users. But the docker part is what ended up impacting me, I thought I did everything I could before hand to pin the drivers and toolkit so the script would succeed and skip the nvidia install but I must have missed something.

Please feel free to reach out, like I said, I have been wrestling with linux, nvidia and all of this for a while now and when it works its great, just needs a little duct tape now and then ;-). -Kreed