Vulkan API backend #5530
Replies: 22 comments 8 replies
|
+1 this should be a priority as it unlocks the use of virtually every device. Consider;
|
|
Please make Vulkan possible. I bought a Strix Halo 128gb just to use Comfy UI, only to find out that I am dying to use Comfy UI on it. |
|
Using Vulkan made installing llama a breeze on my Linux distro that has trouble with ROCm. Wish every AI tool has fist class support for Vulkan first and then targets specific tools. |
|
Reply +1 |
|
+1 |
|
+1 Vulkan would be awesome. |
|
+1 amazing idea |
|
+1 |
|
+1 |
|
+1 |
|
+1 |
|
+1 |
|
+1 |
|
+1 Practically I can't use AI without Vulkan on my machine |
|
+1 |
|
Wouldn't that be on pytorch's side and not really specific to Comfy? Or am I insane? Rocm sucks though and I'm tired of pretending it doesn't. =) |
|
For sure, ticket for it here: pytorch/pytorch#160230 |
|
Llama.cpp has Vulkan capability without need for pytorch (or at least I didn't need to bother with pytorch flatpack LM Studio?). |
|
I believe Llama.cpp cannot be used for diffusion models, there is stable-diffusion.cpp however - which is based on the same underlying libraries and supports Vulkan. My guess is that it's impractical to migrate from pytorch |
|
CC: #5530 (comment) |
|
Sd.cpp's vulkan backend works really well. |
|
CC: #5530 (comment) |
|
+1 |
|
+1 |
|
+1 |
|
I was curious about sd.cpp so I experimented with a reference implementation to test out the performance compared to ROCm. You can download it here: https://github.com/alshdavid/diffusion-vk/releases/tag/0.0.1 It's unoptimised, no attention optimisations but even still it's sometimes faster than ROCm via ComfyUI, and always consistent
Why does AMD even bother with ROCm/HIP |
|
The generative text node really exemplifies the enormous difference in performance |
|
@OrsoEric Is this even a fair comparison? RDNA3 doesnt support fp8 natively and needs to fallback to a slower compute path, which inherently makes it slower than a Q4 quantized model. You are also comparing two different models and model sizes. I'm all for getting more light on vulkan and pushing it further - but this comparison of yours is harmful. Llamacpps vulkan implementation will most likely be faster than pytorches rocm backend, but your statement makes it look worse than it really is. From personal experience, I can say that vulkan overall seems to be 10-15% faster on average compared to ROCm but there are scenarios where ROCm is faster. Generally, ROCm should be faster than Vulkan, but optimizations are tricky. If a contributor pushes out an update to ROCm in pytorch that suddenly boosts its performance past anything we see with any vulkan implementation, a lot of people advocating for vulkan would probably just silently switch back to ROCm. I want to see pytorch vulkan support not because "its faster right now", but because it potentially opens the doors for heterogenic setups that use different GPU types in parallel in the future. |
Yes. I just tested on my 358H laptop, and Qwen 3 4B VL Q4 pulls 21.68 tokens per second with vulkan backend. A laptop with iGPU obliterates a ROCm with a 7900XTX by 50%. It's an humiliating difference. That's the gulf between high performance Vulkan, vs ROCm under windows.
Make pytorch/Vulkan! You already have good people working on the Vulkan/AMD pathways! Don't reinvent the wheels to do pytorch/ROCm/AMD! Until AMD takes ROCm under windows seriously, it's not even a comparison. It's dumb luck when a workload is accelerated under ROCm. I got my 358H with B390. Zimage on 7900XTX pulls 16 seconds, my laptop pulls 50s. That' humiliating for ROCm, the 7900XTX has 9.6X the bandwidth, and 8.5X the raw FP16 FLOPS. And it pulls barely 3X performance. The performance loss of ROCm is inexcusable. And that's not against an high performance CUDA backend, but against they pytorch Intel backend! Intel made better pytorch bindings!!! Vulkan is the only way to get the 7900XTX hardware to shine and put its compute and bandwidth to task. |
and others
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Is a Vulkan backend possible? I use Vulkan on stable-diffusion.cpp and it's significantly faster than CLBlast, CUDA, and ROCm in my testing. I would really like to see it here as Vulkan support is universal and easy, and would help especially with AMD and Intel GPUs (and possibly Qualcomm).
All reactions