Vulkan Video Decoding: Usage Guide and FAQ

[philipl]

Vulkan Video Decoding: Usage Guide and FAQ

On the 28th of May, we reached the significant milestone of finally merging all the required functionality into ffmpeg, libplacebo, and mpv to do end-to-end Vulkan video decoding and presentation. Although the functionality is now all there, we still have a complex landscape in terms of what is supported on what hardware and with what drivers. This document attempts to lay out all the requirements and limitations, so you have a chance of successfully using the feature.

Why should I even care about Vulkan video decoding?

It's a fair question. Right now, the actual functionality you gain access to is not terribly different from what you would get with existing video decoding APIs that ffmpeg and mpv already support. However, Vulkan video decoding has the potential to be a credible cross-vendor, cross-platform API that is well supported, and can work efficiently with Vulkan based filtering and post-processing. In the short term, there will be rough edges, and a narrower set of supported codecs (don't hold your breath for mpeg2 and vp8), but in the future, it will hopefully be the most performant and capable way to do hardware decoding and processing.

Software Requirements

We have finally reached the point where all the required components have made official releases. Ensure you are running with the following releases or newer:

• ffmpeg: 6.1
• libplacebo: v6.292.0
• mpv: 0.37.0

These requirements mean you will likely have to compile everything for yourself (especially on Linux), but Shinchiro's windows builds appear to be functional.

Hardware and Driver Requirements

• Intel on Linux
  • Mesa: mesa-23.2.1
  • A GPU supported by the ANV driver
  • An MR is open for support in the hasvk driver
• AMD on Linux
  • Mesa: mesa-23.2.1
  • A GPU supported by the RADV driver
  • The AMDVLK driver is not known to support video decoding
• Nvidia on Linux or Windows
  • Drivers >= 535.xx
  • Any GPU supported by those drivers
• Nvidia on Linux using Mesa/NVK
  • A build of Mesa with this MR included
  • Hardware supported by NVK
• AMD on Windows
  • Drivers >= 23.9.3
  • RDNA 1/2/3 GPUs
• Intel on Windows
  • Unclear and untested
  • There is evidence that drivers from at least 101.4311 expose the video decoding extensions
  • No idea what hardware is supported
• MacOS (for the record)
  • Although Vulkan on MacOS is supported via MoltenVk, this does not include the decoding extensions, and probably never will. Just keep using videotoolbox.

Usage

The basic command line arguments:

mpv --vo=gpu-next --gpu-api=vulkan --hwdec=vulkan

Depending on your system configuration, you may additionally need to force the right gpu-context:

• Windows: --gpu-context=winvk
• Linux X11: --gpu-context=x11vk
• Linux Wayland: --gpu-context=waylandvk

Capabilities

• Global
  • The Vulkan Video Decoding specs only cover H.264, H.265, AV1, and VP9. Older codecs will require you to use your traditional hardware decoding API (VAAPI, nvdec, dx11va, etc) for now, and possibly forever - depending on whether effort is put into exposing older codecs.
• Intel on Linux
  • You must set ANV_DEBUG=video-decode in your environment to expose video decoding
  • All of H.265, H.265, AV1, and VP9 are supported with new enough Mesa version
  • Vulkan filters are supported from Mesa 24.1 onward.
• AMD on Linux
  • You must set RADV_PERFTEST=video_decode in your environment to expose video decoding
  • All of H.265, H.265, AV1, and VP9 are supported with new enough Mesa version
  • Vulkan filters are supported
• Nvidia on Linux with official drivers
  • Both H.264 and H.265 are supported
  • AV1 is supported with > 550.54.14 drivers
  • VP9 is supported >= 580.xx drivers
  • Vulkan filters are supported
• Nvidia on Linux with NVK
  • Only H.264 is supported right now
  • Vulkan filter support unknown

[Jules-A]

AMD on Windows appears to be 5000/6000 series according to https://www.amd.com/en/support/kb/release-notes/rn-rad-win-22-11-2-vlk-video-code-decode (possibly 7000 works?). It doesn't work on Polaris: 0.339][e][ffmpeg/video] h264: Device does not support the VK_KHR_video_decode_queue extension!

[llyyr]

After bc28dce, you need to patch your Mesa with this PR to get AV1 decoding on radv.

[aufkrawall]

Reliably crashes driver for me with this sample: https://0x0.st/HbHJ.mkv
Other files seem to work for whatever reason (hwdec beeing used confirmed).
Latest git-master of mpv, libplacebo, ffmpeg and radv on 6700 xt.

[philipl]

Sounds like you probably need to file a mesa issue for that - and even if it was an mpv issue, please open a separate one. This FAQ isn't the place to report it.

[cyanreg]

@aufkrawall AMD currently needs https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23227 to avoid crashes, which also fixes quite a lot of bugs

[alakittty]

 (+) Video --vid=1 (*) (hevc 3828x1592 23.976fps)
 (+) Audio --aid=1 --alang=eng (*) (eac3 6ch 48000Hz)
     Subs  --sid=1 --slang=eng (subrip)
     Subs  --sid=2 --slang=eng 'SDH' (subrip)
     Subs  --sid=3 --slang=fre (subrip)
[vo/gpu-next/libplacebo] vk->CreateDevice(vk->physd, &dinfo, PL_VK_ALLOC, &vk->dev): VK_ERROR_OUT_OF_HOST_MEMORY (../../../../../src_packages/libplacebo/src/vulkan/context.c:1308)
[vo/gpu-next/libplacebo] Failed creating logical device!
[vo/gpu-next/libplacebo] Failed initializing vulkan device
[vo/gpu-next] Failed initializing any suitable GPU context!
Error opening/initializing the selected video_out (--vo) device.
Video: no video

what does this error mean, my device has amd and i used the following config
mpv --vo=gpu-next --gpu-api=vulkan --hwdec=vulkan --gpu-context=winvk
mpv was working fine with gpu-next and gpu-api=vulkan till yesterdays' shinchiro release. todays release gave me this error
maybe @shinchiro needs to bump libplacebo and ffmpeg version as per required?

[Jules-A]

AMD on Windows
Special 22.11.2 drivers with video decoding support
Hardware supported by those drivers (documented as 5000/6000 series. Apparently not newer 7000 series)

I don't know if that specific driver is required, they may have been included in 23.2.1 where they added other extensions, the driver notes aren't clear. 7000 series wasn't launched when the 22-11-2 Vulkan video drivers were released so that driver doesn't officially support them. If they did include the video extensions in newer drivers then they may support 7000 series but someone with those GPUs will need to test.

[cyanreg]

Yeah, I've reported to nvidia months ago that their descriptor buffer implementation is broken, but they haven't fixed it yet.
If you want filters working, ask nvidia to fix it.

[philipl]

I found this video will white screen when using ffmpeg vulkan decoding, SW decoding or nvpro vk_video_decoder have no problem, I don't know if this is also NVIDIA's problem (also already reported to NVIDIA)
test.mp4

Github claims you attached an audio-only file...

[llyyr]

That file is also broken on anv, it's glitchy and the decode is very slow while utilizing 100% of Render/3D engine.

[philipl]

That file is also broken on anv, it's glitchy and the decode is very slow while utilizing 100% of Render/3D engine.

It's an HEVC file. My experience on ANV is that any HEVC playback pegs the GPU at 100%

[philipl]

Also:

[ffmpeg/video] hevc: First slice in a frame missing.

This is a highly questionable sample.

[lavilao]

Will this work with hasvk in the future? I tried using it through wine (I dont want to have to compile everything) and got this error [ffmpeg/video] h264: Device does not support the VK_KHR_video_decode_queue extension! even though the output of vulkaninfo is:

vulkaninfo | grep video                                                                     ✔  10s  
	VK_KHR_video_decode_h264                    : extension revision 8
	VK_KHR_video_decode_queue                   : extension revision 7
	VK_KHR_video_queue                          : extension revision 8
			videoCodecOperations:
			videoCodecOperations: count = 1

[CharlesMengCA]

NVIDIA 1650 Super, Tested on windows with: mpv --vo=gpu-next --gpu-api=vulkan --hwdec=vulkan --gpu-context=winvk
5 videos played OK.

This video has problem: https://4kmedia.org/sony-whale-in-tonga-hdr-uhd-4k-demo/
Switched back to hwdec=nvdec, the problem is gone.

[24fpsDaVinci]

for fedora, you need to edit mesa.spec and rebuild with codecs enabled -D video-codecs=h264dec,h264enc,h265dec,h265enc,vc1dec, codecs is disabled by default.

rpmfusion-infra/fedy#110

https://themaister.net/blog/2023/01/05/vulkan-video-shenanigans-ffmpeg-radv-integration-experiments/

https://forum.endeavouros.com/t/compiling-mesa-22-2-with-codec-support-the-easy-way/30390

[rathann]

No need to recompile, install mesa-vulkan-drivers-freeworld from https://rpmfusion.org/ instead.

[CounterPillow]

opengl has no video decoder interface. You're talking about the rendering backend, not the hardware decoder API. Also who asked

[Isaacx123]

Working fine on nvidia r570 drivers and ffmpeg 7.1, decoding H265 8/10-bit, H264 and AV1 without issue, but the frame timings are significantly worse when compared to nvdec, 2ms vs 7ms when decoding a 4K AV1 video.

[Isaacx123]

The 570.124.04 driver has fixed the frame timing difference between vulkan and nvdec, now they are pretty much the same.

[ralmachado]

Even though the AMD RADV iGPU is enumerated first, mpv will default to using the discrete NVIDIA gpu with hwdec=auto:

[vo/gpu/libplacebo] Probing for vulkan devices:
[vo/gpu/libplacebo] Spent 16.978 ms enumerating physical devices
[vo/gpu/libplacebo]     GPU 0: AMD Radeon Graphics (RADV RENOIR) v1.4.305 (integrated)
[vo/gpu/libplacebo]            uuid: 00:00:00:00:06:00:00:00:00:00:00:00:00:00:00:00
[vo/gpu/libplacebo]     GPU 1: NVIDIA GeForce RTX 3050 Laptop GPU v1.4.303 (discrete)
[vo/gpu/libplacebo]            uuid: 71:84:38:67:B8:C7:AB:F9:F5:9C:4A:30:EF:54:9D:62
[vo/gpu/libplacebo]     GPU 2: llvmpipe (LLVM 19.1.7, 256 bits) v1.4.305 (software)
[vo/gpu/libplacebo]            uuid: 6D:65:73:61:32:35:2E:30:2E:30:00:00:00:00:00:00
[vo/gpu/libplacebo] Vulkan device properties:
[vo/gpu/libplacebo]     Device Name: NVIDIA GeForce RTX 3050 Laptop GPU
[vo/gpu/libplacebo]     Device ID: 10de:25a2
[vo/gpu/libplacebo]     Device UUID: 71:84:38:67:B8:C7:AB:F9:F5:9C:4A:30:EF:54:9D:62
[vo/gpu/libplacebo]     Driver version: 8e958400
[vo/gpu/libplacebo]     API version: 1.4.303
...
[vd] Codec list:
[vd]     h264 - H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
[vd]     h264_v4l2m2m (h264) - V4L2 mem2mem H.264 decoder wrapper
[vd]     h264_qsv (h264) - H264 video (Intel Quick Sync Video acceleration)
[vd]     libopenh264 (h264) - OpenH264 H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
[vd]     h264_cuvid (h264) - Nvidia CUVID H264 decoder
[vd] Opening decoder h264
[vd] Looking at hwdec h264-nvdec...
[vo/gpu] Loading hwdec drivers for format: 'cuda'
[vo/gpu] Loading hwdec driver 'cuda'
[vd] Trying hardware decoding via h264-nvdec.
[vd] Pixel formats supported by decoder: vdpau vulkan cuda vaapi yuv420p
[vd] Codec profile: High (0x64)
[vd] Requesting pixfmt 'cuda' from decoder.
[vd] Selected decoder: h264 - H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
[vd] Pixel formats supported by decoder: vdpau vulkan cuda vaapi yuv420p
[vd] Codec profile: High (0x64)
[vd] Requesting pixfmt 'cuda' from decoder.
Using hardware decoding (nvdec).
[vd] Decoder format: 1920x1080 cuda[nv12] bt.709/bt.709/bt.1886/limited/auto CL=mpeg2/4/h264 crop=1920x1080+0+0
[vd] Using container aspect ratio.
[vo/gpu] Loading hwdec drivers for format: 'cuda'
AO: [pipewire] 48000Hz 5.1(side) 6ch floatp
VO: [gpu] 1920x1080 cuda[nv12]

I can manually set the vulkan device to the AMD RADV, and it will use vaapi instead of nvdec, but isn't mpv supposed to use the first device enumerated?

[ralmachado]

Another weird behaviour, both with OpenGL and Vulkan, and forcing vulkan-device to the iGPU, mpv wakes up the dGPU from D3cold to D0, and the GPU only goes back to D3cold after closing mpv.

VLC keeps the dGPU in D3cold.

Edit: it seems like the vo tries to use nvdec even though its an AMD gpu, which makes no sense to me. A hacky solution is to set hwdec=vaapi,auto in mpv.conf, which still wakes the dGPU for a few moments, but eventually allows it to go back to D3cold.

[llyyr]

How is this related to vulkan hwdec? mpv will use the dGPU if available, regardless of the order they're enumerated in. If you want it to use a specific gpu then put it in mpv.conf. You can use --vulkan-device=help to figure out the specific device name you need to put.

it seems like the vo tries to use nvdec even though its an AMD gpu

This is how auto works, it probes for all available hwdec APIs and nvdec is checked first since if available, the user probably wants to use that instead of the others. Specify a specific API instead.

regardless open a new issue/discussion, since this is unrelated to hwdec=vulkan

[ralmachado]

mpv uses the iGPU with OpenGL, and that is not what the manpage for vulkan-device says. Quoting:

--vulkan-device=<device name|UUID>
              The name or UUID of the Vulkan device to use for rendering and presen‐
              tation. Use --vulkan-device=help to see the list of available  devices
              and  their  names and UUIDs. If left unspecified, the first enumerated
              hardware Vulkan device will be used.

Output of mpv --vulkan-device=help lists AMD first:

Available vulkan devices:
  'AMD Radeon Graphics (RADV RENOIR)' (GPU 0, PCI ID 1002:1638, UUID 00000000-0600-0000-0000-000000000000)
  'NVIDIA GeForce RTX 3050 Laptop GPU' (GPU 1, PCI ID 10de:25a2, UUID 71843867-b8c7-abf9-f59c-4a30ef549d62)
  'llvmpipe (LLVM 19.1.7, 256 bits)' (GPU 2, PCI ID 10005:0, UUID 6d657361-3235-2e30-2e30-000000000000)

[Kimiblock]

There’s a known problem that NVIDIA’s Vulkan layer selects the discrete GPU by default

[Vamp898]

Hi there,

i have an AMD GPU (RADV) that is builtin into my Ryzen 7 5700G and i noticed that mpv now uses waylandvk by default on my system where it used just wayland as context until recently.

I noticed it, because it is horribly slow, its barely usable. Using --profile=best causes most of the frames to be dropped and using profile=fast gets me through with 90% GPU usage and a few frame drops here and there.

For comparison with --gpu-context=wayland, profile=fast uses like 5~10% GPU and profile=best works without frame drops at ~90% GPU usage.

So i put gpu-context=wayland into my mpv.conf now but i am wondering what causes this insane frame drop.

Its a FullHD Video i watch on a 4K Screen (so it is using the upscaling), H.264 (with vaapi decoding) and a pretty low bit-rate. Like 3mbps or so (YouTube Quality). So not sure what exactly is the issue here, but waylandvk is not usable for me, at all.

[kasper93]

Do you use hwdec? And if so, does disabling/enabling it changes overall performance? Also this probably should be a separate issue with a log file to help us troubleshoot. Vulkan shouldn't be significantly slower than opengl, but there may be some specificities in the driver that makes Vulkan hwdec not work well for example with native rendering, but we have not enough Info to know that.

[christopherplaysminecraft]

Could someone that knows a bit more about this than me figure out if AMD GPUs still require RADV_PERFTEST? I do not know where to look at to find that information. According to another comment in this discussion it depends on a certain Mesa version, and also on the VCN version. So it would be nice to know which Mesa versions with which VCN versions don't require the flag to work.

The Arch Wiki does not mention RADV_PERFTEST itself as a requirement to get Vulkan Video Decoding to work, only Mesa 25.1 or higher. (On the KDE page under the HDR section)

[christopherplaysminecraft]

Guess what specifically?

I will be getting an RX 9070 XT.

[shmerl]

So, how hard is it to set the variable? Not sure what you are even worrying about.

[Glog78]

echo "RADV_PERFTEST: " $RADV_PERFTEST && vulkaninfo 

https://paste.rs/9PywJ

Maybe someone help me but for me and the 9070 xt the extensions aren't showing ... even with RADV_PERFTEST set :(

[llyyr]

You need to set RADV_PERFTEST=video_decode, not set it to a blank value.

[Glog78]

let me correct what i wrote so it's easier to understand the echo and the vulkaninfo is going into the pasti so you see how $RADV_PERFTEST is set and what is the current vulkaninfo ;) Sorry i forget to mark code. Thank you for taking your time and trying to help.

[amayra]

does glsl shaders like KrigBilateral and SSimDownscaler support in vulkan backend

or i should keep using openGL to use them

[CounterPillow]

this has nothing to do with vulkan decoding.

[random327]

I'm trying to use vulkan video instead of nvdec due to it being inefficient, forcing the GPU into a high power profile.
But mpv doesn't even start at all with any video i try (h264, h265, av1):

[ffmpeg] av_log callback called with bad parameters (NULL AVClass).
[ffmpeg] This is a bug in one of FFmpeg libraries used.
[ffmpeg] Unable to submit command buffer: VK_ERROR_DEVICE_LOST
[ffmpeg/video] hevc: get_buffer() failed
[ffmpeg/video] hevc: thread_get_buffer() failed
[ffmpeg/video] hevc: Error parsing NAL unit #0.
Error while decoding frame (hardware decoding)!

Specs are: 5070ti, Version: 570.144, Arch Linux, Wayland, Plasma, Open GPU Kernel Modules.

Hope someone can help, unsure if this is a driver thing with the new 50 series..

mpv --gpu-debug --msg-level=all=debug --no-config --vo=gpu-next --gpu-api=vulkan --hwdec=vulkan --gpu-context=waylandvk --log-file=log.txt t.mkv

log.txt

[CounterPillow]

@adolfotregosa please don't use vdpau in any case. Unless something drastically changed in the past year, nvdec should always be preferred for nvidia, as vdpau is on its deathbed.

[adolfotregosa]

@adolfotregosa please don't use vdpau in any case. Unless something drastically changed in the past year, nvdec should always be preferred for nvidia, as vdpau is on its deathbed.

https://forums.developer.nvidia.com/t/wayland-support-for-the-575-release-series/333827

"""VDPAU support on Wayland."""

Nvdec uses more power because gpu P state stays locked in P3. Vdpau let's gpu P state go higher using less power.

[random327]

NVIDIA Driver version 580.76.05 appears to have finally fixed this, i can now use vulkan hwdec with mpv.

https://www.nvidia.com/en-us/drivers/details/252613/

Fixed a bug that could cause Vulkan applications to hang when destroying swapchains after a lost device event.

[adolfotregosa]

Thank you for sharing. Even AV1 decoding works with vulkan ! It still fails with vdpau-copy but it's progress.

[philipl]

To avoid the power wastage, you can use this: https://gist.github.com/Ristovski/c81502f850ba095320353ec2094b14cf

[mcfd]

ANV_VIDEO_DECODE=1 to ANV_DEBUG=video-decode
https://bbs.archlinux.org/viewtopic.php?id=305668

[mervn]

@philipl is it possible to get an update in the main post about this? I can see this tripping a lot of people up.

For anyone curious, the list of Mesa environment vars are at https://docs.mesa3d.org/envvars.html ; with Intel (ANV) in particular at https://docs.mesa3d.org/envvars.html#anvil-anv-driver-environment-variables

[Archer2011]

Khronos Announces Vulkan Video Decode VP9 Extension
https://www.khronos.org/blog/khronos-announces-vulkan-video-decode-vp9-extension

[kasper93]

should already work for amd gpus if you're building mesa from main branch

What do you mean by that? Work where?

[llyyr]

As in, radv supports that extension now https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/35398

ffmpeg doesn't have a patch yet, but gstreamer does https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/9204

[kasper93]

As in, radv supports that extension now https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/35398

ffmpeg doesn't have a patch yet, but gstreamer does https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/9204

Sir, this is an mpv repository. I doubt anything will work until there are patches in both ffmpeg and mpv to enable VP9 extension.

[CounterPillow]

mpv should move to gstreamer anyway

[philipl]

With new enough Vulkan headers, ffmpeg, and mpv, it should work today. I don't have the headers and haven't bothered installing them, so can't confirm.

[DonKatsu]

I can't get vulkan decode working on my 9070.
I put RADV_PERFTEST=video_decode in my environment, but vulkaninfo | grep video still returns nothing and mpv gives [ffmpeg/video] hevc: Device does not support the VK_KHR_video_decode_queue extension! with hwdec=vulkan.
I'm on CachyOS with kernel 6.15.4, mesa 25.1.4, mpv 0.40.0.

[DonKatsu]

Eh, found this after the fact. Guess it's either wait for mesa 25.2 or switch to mesa-git.
https://www.phoronix.com/news/Mesa-25.2-VCN5-RDNA4-Video

[cih89]

I'm on Intel (i3 1215u iGPU)+Arch Linux with current Mesa. Vulkan decoding works, but fast forwarding/skipping any H265 file makes it crash with this error:

[ffmpeg/video] hevc: hardware accelerator failed to decode picture
Error while decoding frame (hardware decoding)!
[ffmpeg/video] hevc: Unable to perform query: VK_ERROR_DEVICE_LOST!
[ffmpeg/video] hevc: hardware accelerator failed to decode picture
Error while decoding frame (hardware decoding)!
[ffmpeg/video] hevc: Unable to perform query: VK_ERROR_DEVICE_LOST!
[ffmpeg/video] hevc: hardware accelerator failed to decode picture
Error while decoding frame (hardware decoding)!
[ffmpeg] vk: Unable to submit command buffer: VK_ERROR_DEVICE_LOST
[vo/gpu-next/libplacebo] vkQueueSubmit2: VK_ERROR_DEVICE_LOST (../src/vulkan/command.c:504)
[vo/gpu-next/libplacebo] Retrieving query pool results: VK_ERROR_DEVICE_LOST (../src/vulkan/gpu.c:105)
[vo/gpu-next/libplacebo] vkQueueSubmit2: VK_ERROR_DEVICE_LOST (../src/vulkan/command.c:504)
[vo/gpu-next/libplacebo] Retrieving query pool results: VK_ERROR_DEVICE_LOST (../src/vulkan/gpu.c:105)
[vo/gpu-next/libplacebo] vkQueueSubmit2: VK_ERROR_DEVICE_LOST (../src/vulkan/command.c:504)
[vo/gpu-next/libplacebo] Retrieving query pool results: VK_ERROR_DEVICE_LOST (../src/vulkan/gpu.c:105)

H264 files work without issues.

[kasper93]

Report it to mesa.

[sandersr]

I was wondering what else is needed for this to work with virtio-gpu via venus driver