GPU VRAM detection

[mudler]

After #7891 and #7907, which has obviously very good upsides such as smaller images, and simplified UX (one image to rule them all), comes also with some drawbacks.

One of the few is that now we can't rely anymore on the GPU vendor binaries (such as rocm-smi, vulkaninfo etc) to be present in the container image. This leaves the user with 3 options I can think of at the moment:

• Pre-execute a LocalAI script on start to install the required tools, or build a container image of LocalAI with the tools required included
• Mount the binaries in the image manually from the host
• Run LocalAI outside the container image in the host (with the tools installed)

This issue is mainly as a discussion point on how to tackle this. There is some trade-offs here, between the container image sizes, UX and VRAM monitoring features. We can of course still build separate container images for each of the GPU vendors. This might minimize impact of this, but would be nice to reduce the strain on the CI.

[mudler]

For the moment restored the workflow to keep building images here: #7910 #7911.

However, master images works also with Nvidia GPUs without any issues (I've tested with the DGX Spark).

master images with Rocm, vulkan and intel won't display GPU usage.

[richiejp]

What about create a backend that exports VRAM info instead of doing inference? Then suggest the user installs that next to the RAM usage if they have a GPU.

[Expro]

UX + feature > container image size. Non-container use cases are no go for many people (myself included). Polluting OS with workload-specific binaries and configuration is something many people would like to avoid.

Also, from perspective of someone who mixes various GPU generations and often vendors - having one image that has both ROCm + Vulkan would be great improvement over separate image for ROCm and Vulkan. From that perspective, we might as well throw in CUDA and have one unified image with everything, even if it weights 30GB. Most models weights ten times of that (per model!) anyone looking for AI have to prepare storage anyway.