What's preventing the use of GPU in docker containers?

[AnkushMalaker]

I was able to give access to my GPU by modifying the docker-compose file and using nvidia/cuda:12.1.1-runtime-ubuntu22.04 as the base image.
torch.cuda.is_available() is also True.
What's preventing the LLM from utilizing the GPU that'd work otherwise with local install, according to the offline + gpu instructions?

[debanjum]

That's awesome to hear! What OS are you running on? And do you see GPU utilization spike when offline chat model responds?

Few things that needed to be done were:

1. Test offline chat models utilize GPU on Ubuntu, Windows, Mac when using Khoj docker image built on Nvidia base
2. Add/update Dockerfile, Github workflow to publish (separate?) Nvidia based Khoj docker images, once step 1 works

[debanjum]

The main thing to verify was that the underlying library we use, llama.cpp, is able to see+use the system GPU on the major operating systems when used via Docker

[AnkushMalaker]

I'm running windows with docker on wsl backend with RTX 4090.
No, no spike. All processors were 100% indicating full CPU use.

[AnkushMalaker]

I'm pretty sure llama.cpp can see and use the system GPU.
And I think I can give it a shot to try to run it on Docker on windows which is major enough?
I also have an Ubuntu system so I can see about it running on both which is also super important to me lol

[AnkushMalaker]

Point me to where int the code we make decisions about where to run the LLM? (ie CPU vs GPU)

[rehandaphedar]

It is

khoj/src/khoj/utils/helpers.py

Line 297 in 8920409

def get_device() -> torch.device:

[rehandaphedar]

Something information that might be useful - I used another program fish which supports using GPU for training+inference in Docker.

It seems to come down to:

• Installing NVidia Container Toolkit
• Passing --gpus all when running Docker

I tried doing the same thing. However, torch.cuda.is_available() returned False.