GPU not being used in Jan

[ux-han]

I'm having some trouble with my Nvidia GPU not being utilized properly in the Jan app.
When I try to load and run AI models in the Jan app, I notice that my Nvidia GeForce GTX 1650 GPU (with 4GB of VRAM) doesn't seem to be getting used.

jan.mp4

[ux-han]

Based on the details you provided, it seems the problem may be related to the ngl (number of GPU layers) setting for the AI models you're trying to load.
The default ngl setting for many models is 33, which appears to be exceeding the 4GB VRAM capacity of your Nvidia GTX 1650 GPU. That's why you had to manually reduce the VRAM usage down to around 25GB or lower to get the models to start up and use the GPU.
The ngl setting determines how many GPU layers the model will use during inference. Higher ngl values generally require more VRAM, which is why the 20GB model you tried to load wouldn't start up on your GPU.

To resolve this issue, we recommend trying the following:

• InThreads tab in Jan app, open Threads Settings in the right sidebar, and look for an option to adjust the ngl or "number of GPU layers" parameter. Try reducing this value to 12 or lower.
• If adjusting the ngl setting doesn't work, you can also try loading the models one by one and monitoring the VRAM usage. This will help you identify the specific models that are exceeding your GPU's memory capacity.
• As a last resort, you may need to consider using a GPU with more VRAM, such as an Nvidia RTX 3060 or higher, to be able to run the full set of 41 models without issues.