Unraid CUDA hardware-acceleration won't work (GTX 980)

[DorCoMaNdO]

Per the Immich documentation (link), the minimum supported CUDA capabilities are 5.2, and CUDA version 12.3 (driver 545, I'm using 580, latest for this GPU), which the GTX 980 supports.
I've passed all the standard hardware-acceleration variables and the --runtime=nvidia parameter, all of which works perfectly fine in other containers (i.e. Plex), and it appears to be working for Immich (the GPU Statistics Unraid plugin shows it as an "active app"), I've also run python3 -c "import onnxruntime as ort; print(ort.get_available_providers())" and it returns ['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider'].

However, once any hardware-acceleration operation attempts to run, it fails with the message below:

Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running Gather node.
Name:'/token_embedding/Gather'
Status Message: CUDA error cudaErrorNoKernelImageForDevice:no kernel image is available for execution on the device

[hydazz]

I believe this is the upstream issue
immich-app/immich#25732

does not appear to be a problem with this image

[DorCoMaNdO]

I guess that's true, and I saw your reply in #665, I've recently switched from the docker mod to a separate Valkey docker, given that ML in this "monolithic" container is non-functional for me, do I gain anything by using this container over the regular one (and adding an older ML container that would support my card)? If not, is it as straightforward as switching to ghcr.io/immich-app/immich-server? Alternatively, is it possible to run two ML/transcoding workers (built in as cpu worker + external as gpu worker)?