-
Notifications
You must be signed in to change notification settings - Fork 16
Description
Describe what problem this solves and why this would be valuable to many users
As far as I can tell, the docker compose.yaml example is hardcoded for nvidia GPUs due to:
NVIDIA_VISIBLE_DEVICES: "all"
NVIDIA_DRIVER_CAPABILITIES: "compute,utility"
## Share host devices to enable CPU/GPU hardware acceleration (optional):
# devices:
# - "/dev/dri:/dev/dri"
# - "/dev/nvidia0:/dev/nvidia0"
# - "/dev/nvidiactl:/dev/nvidiactl"
# - "/dev/nvidia-modeset:/dev/nvidia-modeset"
# - "/dev/nvidia-nvswitchctl:/dev/nvidia-nvswitchctl"
# - "/dev/nvidia-uvm:/dev/nvidia-uvm"
# - "/dev/nvidia-uvm-tools:/dev/nvidia-uvm-tools"
deploy:
resources:
reservations:
devices:
- driver: "nvidia"
capabilities: [ gpu ]
count: "all"
This may mean that other GPUs are not supported which leaves out quite a few users. (does this mean everything is CPU run?)
I looked a bit into the docker deploy but don't really get how docker knows to reserve the right nvidia resources
Describe the solution you'd like
Add support for intel GPUs similar to what https://github.com/eleiton/ollama-intel-arc does which I believe is done using https://github.com/intel/ipex-llm
Describe alternatives you've considered
https://github.com/eleiton/ollama-intel-arc does provide an Ollama for Intel GPU service which can be used with the remote llama integration. While I have tested it individually, I have not yet tested it with photoprism-vision.
It may also be that the ollama performance is on par or better than the native models listed (Kosmos-2/VIT-GPT2/BLIP) in which case this may not be a huge issue as ollama-intel-arc could be used as a potential workaround