How to make Ollama utilize GPU resources for running large models?

[SimonXu666j]

After downloading Ollama and LoBaChat from the app store, when using Ollama-deployed large models through LoBaChat, I noticed Ollama only utilizes CPU instead of GPU. How can I make Ollama leverage GPU computing? Upon entering the Ollama pod, I found no GPU-related driver information, even though NVIDIA drivers and CUDA are properly installed on the host machine.

[eball]

You can try to execute the command

kubectl get pods -A | grep -i nvidia

to check if the Nvidia device plugin pod is running properly.

if not , you can try to execute the command

sudo olares-cli gpu install && sudo olares-cli gpu enable

and re-install the ollama

Normally Olares will install the GPU driver automatically, also you can provide us the install logs in ${HOME}/.olares/logs to help us find out the problem

THX

[SimonXu666j]

root@localhost:/.olares/versions/v# ll /root/.olares/versions/v
total 16
drwxr-xr-x 4 root root 4096 Mar 27 15:50 ./
drwxr-xr-x 4 root root 4096 Mar 27 15:50 ../
drwxr-xr-x 2 root root 4096 Mar 27 15:50 cli/
drwxr-xr-x 2 root root 4096 Mar 27 15:50 logs/
root@localhost:/.olares/versions/v# cd
root@localhost:~# sudo olares-cli gpu install && sudo olares-cli gpu enable
current: root
2025-03-27T15:53:15.638+0800 unable to read manifest, open /root/.olares/versions/v/installation.manifest: no such file or directory
2025-03-27T15:53:15.639+0800 [FATAL] open /root/.olares/versions/v/installation.manifest: no such file or directory
bytetrade.io/web3os/installer/pkg/pipelines.InstallGpuDrivers
/home/runner/work/Installer/Installer/pkg/pipelines/gpu_install.go:32
bytetrade.io/web3os/installer/cmd/ctl/gpu.NewCmdInstallGpu.func1
/home/runner/work/Installer/Installer/cmd/ctl/gpu/install.go:17
github.com/spf13/cobra.(*Command).execute
/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:987
github.com/spf13/cobra.(*Command).ExecuteC
/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115
github.com/spf13/cobra.(*Command).Execute
/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039
main.main
/home/runner/work/Installer/Installer/cmd/main.go:15
runtime.main
/opt/hostedtoolcache/go/1.22.4/x64/src/runtime/proc.go:271
and how to re-install the ollama

[eball]

Sorry, it looks like your Olares was installed by another user in linux system.
The command was executed failed

you can try to find out the user that used to install the Olares and re-execute the command

sudo olares-cli gpu install --base-dir /<user's home>/.olares --version 1.11.5 && \
sudo olares-cli gpu enable --base-dir /<user's home>/.olares --version 1.11.5

And then you can uninstall the ollama in the Market

and install it again.

[SimonXu666j]

execute the command
olares-cli gpu install --base-dir /root/.olares/ --version 1.11.5 && sudo olares-cli gpu enable
it wokrs, thanks