I have multiple GPUs of different architectures in the same machine (for example, a laptop with an intel iGPU + nvidia dGPU), where I want to run Vulkan based llama.cpp in iGPU (where I have 96 Gb of shared memory, but it is awfully slow) for one model, and cuda based llama in dGPU (where I have only 4 GB of memory, and I can only feasibly run nemotron-4b for light tasks)
I would be very glad if I could do this within the lms.
Maybe the selection of runtime can be carried to model load, and can be selected per model instead of a global setting, sparkrun style?
I have multiple GPUs of different architectures in the same machine (for example, a laptop with an intel iGPU + nvidia dGPU), where I want to run Vulkan based llama.cpp in iGPU (where I have 96 Gb of shared memory, but it is awfully slow) for one model, and cuda based llama in dGPU (where I have only 4 GB of memory, and I can only feasibly run nemotron-4b for light tasks)
I would be very glad if I could do this within the lms.
Maybe the selection of runtime can be carried to model load, and can be selected per model instead of a global setting, sparkrun style?