Video Encode/Decode

[Daedie-git]

The open encode/decode enticed me to try and get VK hardware decoding working in my software (until now, I only had it on the D3D12 side). I expect this will need further changes, just let me know.
I wasn't sure what the right way to handle the D3D12 CommandBuffer was, I settled on the variant for now.

Summary

Adds hardware video queue support and a minimal native-backed video encode/decode extension API.

This branch makes video-capable queues visible through NRI, allows D3D12/Vulkan video command buffers to be created and submitted, adds
basic multi-planar video format support, and exposes NRIVideo entry points for issuing native encode/decode commands.

Changes

Public API

• Adds QueueType::VIDEO_DECODE
• Adds QueueType::VIDEO_ENCODE
• Adds multi-planar 4:2:0 formats:
  • G8_B8R8_2PLANE_420_UNORM
  • G10X6_B10X6R10X6_2PLANE_420_UNORM_3PACK16
  • G16_B16R16_2PLANE_420_UNORM
• Adds explicit image plane bits for multi-planar images:
  • PLANE_0
  • PLANE_1
  • PLANE_2
• Adds Include/Extensions/NRIVideo.h
  • VideoInterface
  • CmdDecodeVideo
  • CmdEncodeVideo
  • native D3D12/Vulkan encode/decode descriptor structs

Adapter and Queue Discovery

• Reports D3D12 video decode/encode queue availability when ID3D12VideoDevice is available
• Adds Vulkan queue family selection support for video queues
• Uses VkQueueFamilyVideoPropertiesKHR to ensure Vulkan video queues also expose compatible codec operations
• Reuses queue family create infos when multiple NRI queue types map to the same Vulkan queue family
• Tracks queue storage for the new video queue types in D3D12 and Vulkan devices

D3D12 Backend

• Maps NRI video queues to:
  • D3D12_COMMAND_LIST_TYPE_VIDEO_DECODE
  • D3D12_COMMAND_LIST_TYPE_VIDEO_ENCODE
• Refactors CommandBufferD3D12 to hold graphics, video decode, or video encode command lists
• Supports begin/end/reset/close for D3D12 video command lists
• Submits video command lists through the existing queue submission path
• Adds CmdDecodeVideo via ID3D12VideoDecodeCommandList::DecodeFrame
• Adds CmdEncodeVideo via ID3D12VideoEncodeCommandList2::EncodeFrame
• Updates native command buffer documentation to return ID3D12CommandList*, since video command lists are not graphics command lists

Vulkan Backend

• Adds video-related desired device extensions, including:
  • VK_KHR_video_queue
  • VK_KHR_video_decode_queue
  • H.264/H.265/AV1 decode extensions
  • VK_KHR_video_encode_queue
  • video maintenance extensions
  • sampler YCbCr conversion
• Adds Vulkan format mappings for the new multi-planar formats
• Adds Vulkan plane aspect mapping for explicit plane views
• Allows texture views to select explicit multi-planar aspects
• Resolves vkCmdDecodeVideoKHR and vkCmdEncodeVideoKHR
• Adds CmdDecodeVideo / CmdEncodeVideo forwarding to Vulkan command buffers

Validation / Interface Wiring

• Wires VideoInterface through nriGetInterface
• Adds DeviceBase::FillFunctionTable(VideoInterface&)
• Adds backend FillFunctionTable(VideoInterface&) implementations
• Adds validation-layer storage and forwarding for the video interface
• Reports unsupported when the backend/device does not expose video decode or encode queues

API Shape

The new video API is intentionally small and native-backed.

NRI handles queue discovery, command buffer ownership, function table exposure, and command dispatch. Codec/session setup remains
backend-native (initially):

• D3D12 callers pass native decoder/encoder objects and D3D12 argument structs
• Vulkan callers pass native VkVideoDecodeInfoKHR / VkVideoEncodeInfoKHR payloads

Validation

Built successfully with:

• Vulkan/validation-focused configuration
• D3D12-focused configuration
• small encode -> decode unit test in my own repo

[dzhdanNV]

Wow! Magnificent! Thanks a ton for bringing this work to life! I will try to review later today.

[dzhdanNV]

Question 1

Should VideoDecodeD3D12Desc, VideoEncodeD3D12Desc, VideoDecodeVKDesc and VideoEncodeVKDesc be moved into corresponding wrappers and the underlying objects created via NRI? (not a pro in video APIs yet)

Question 2

Should we add more formats? We should invent better names :) DXGI has short names, but they are unclear (seems to be a good start):

    G8_B8R8_2PLANE_420_UNORM,
    G10X6_B10X6R10X6_2PLANE_420_UNORM_3PACK16,
    G16_B16R16_2PLANE_420_UNORM,

Question 3

No D3D11 support? Probably not needed, but, IM, nice to have :)

(...to be continued)

[Daedie-git]

Q1
I'll get back to you on this.

Q2:
How about this

NV12_UNORM
P010_UNORM
P016_UNORM

Q3
D3D11 video semantics work very differently. So regardless if it is added or not, I would do that in a separate PR.

[dzhdanNV]

Q2

+1, clarifying comments may be added if needed. They are pretty standard.

Q3

No objections here (assuming the interface is implementable in D3D11)

[dzhdanNV]

Do we need other formats, like DXGI_FORMAT_Y410 (10:10:10)?

[dzhdanNV]

Please, add this link somewhere in the beginning of NRIVideo.h: https://learn.microsoft.com/en-us/windows/win32/medfound/recommended-8-bit-yuv-formats-for-video-rendering

[dzhdanNV]

I'm ready to merge it after you are done with polishing. Just let me know. Further improvements can be made in main. Can we create a minimalistic sample in https://github.com/NVIDIA-RTX/NRISamples for this?

I haven't deeply analyzed the implementation, but the API additions are nice and clear.

Also I think that these formats:

    // Depth-stencil (as a shader resource view)
    R24_UNORM_X8,       // .x - depth   // + . . . . . . . . . . . . . . .
    X24_G8_UINT,        // .y - stencil // + . . . . . . . . . . . . . . .
    R32_SFLOAT_X8_X24,  // .x - depth   // + . . . . . . . . . . . . . . .
    X32_G8_UINT_X24     // .y - stencil // + . . . . . . . . . . . . . . .

can be removed and readonlyPlanes can be used instead explicitly. readonlyPlanes are needed for RTV and actually for SRV (you already use it for video). It's a bit of a breaking change, but, IMO, a step towards better API. What do you think? Just asking for your opinion, I will implement it myself.

[Daedie-git]

I'm ready to merge it after you are done with polishing. Just let me know. Further improvements can be made in main. Can we create a minimalistic sample in https://github.com/NVIDIA-RTX/NRISamples for this?

Definitely. I'll look into it as soon as I can.

I haven't deeply analyzed the implementation, but the API additions are nice and clear.

Also I think that these formats:

    // Depth-stencil (as a shader resource view)
    R24_UNORM_X8,       // .x - depth   // + . . . . . . . . . . . . . . .
    X24_G8_UINT,        // .y - stencil // + . . . . . . . . . . . . . . .
    R32_SFLOAT_X8_X24,  // .x - depth   // + . . . . . . . . . . . . . . .
    X32_G8_UINT_X24     // .y - stencil // + . . . . . . . . . . . . . . .

can be removed and readonlyPlanes can be used instead explicitly. readonlyPlanes are needed for RTV and actually for SRV (you already use it for video). It's a bit of a breaking change, but, IMO, a step towards better API. What do you think? Just asking for your opinion, I will implement it myself.

I like that.

[Daedie-git]

Do we need other formats, like DXGI_FORMAT_Y410 (10:10:10)?

The current set targets the common 4:2:0 format family needed by the decode/encode path: NV12, P010, P016. Y410 is packed 4:4:4 and would add support for it in a follow-up if the need arises.

[vertver]

I don't like that you're using native API structures in NRI public API. Wouldn't be logical to implemented it as a translation layer, not in the NRIWrapper* way?

[dzhdanNV]

I like that.

@Daedie-git I have implemented it. Please, update your code to the latest since it's a prerequisite for the video support. Please, note that I have removed PR #222 since the code is completely different for these lines. Feel free to adjust, re-add or discuss. Thanks in advance! We are moving forward!

I don't like that you're using native API structures in NRI public API. Wouldn't be logical to implemented it as a translation layer, not in the NRIWrapper* way?

Just echoing already said, I absolutely agree with this. GAPI specific structs must be moved to corresponding NRIWrapperX extension and NRIVideo must get an abstraction. This is what we should focus on.

[Daedie-git]

I like that.

@Daedie-git I have implemented it. Please, update your code to the latest since it's a prerequisite for the video support. Please, note that I have removed PR #222 since the code is completely different for these lines. Feel free to adjust, re-add or discuss. Thanks in advance! We are moving forward!

I don't like that you're using native API structures in NRI public API. Wouldn't be logical to implemented it as a translation layer, not in the NRIWrapper* way?

Just echoing already said, I absolutely agree with this. GAPI specific structs must be moved to corresponding NRIWrapperX extension and NRIVideo must get an abstraction. This is what we should focus on.

Indeed. I have addressed this locally already. I'll rebase it on your change and push it soon.

[Daedie-git]

I wrapped up my changes.

[Daedie-git]

I found some issues, so I will follow up with another commit.

[Daedie-git]

Additional question: For the sake of testing this, I've been building up quite the test suite in my own repo. Do you want to have it in some form?
Apart from the samples you already asked for.

[dzhdanNV]

I'm with Codex now. I'm getting up to speed, but Codex has already proven to be extremely helpful. Last time I addressed many D3D12 issues, but not all of them. I haven't touched VK yet. I will get back to this on Monday. I'm sure Codex will help to accelerate. My plan is to fix NRI related issues, not video related issues to avoid introducing unnecessary noise to your work.

[Daedie-git]

I've managed to chew through most of the missing features.
I've skipped AV1 segmentation on purpose, because my GPU doesn't seem to support it. So I have no way of properly testing it.

I'll also be upgrading the video sample project to the new reality soon. To make sure all the paths work beyond what my test suite is already covering.

[Daedie-git]

Also did a round of bugfixes. have all 6 codecs working again in the sample.
Next I want to upgrade the sample a bit and improve the UI a bit as well.

[dzhdanNV]

Please, add big TODOs right into NRIVideo.h.

[dzhdanNV]

Please, add big TODOs right into NRIVideo.h.

Just spotted VIDEO_TODO.md. It's even better. Keep it.

[Daedie-git]

Semi-update: I've been working on building a new graphics stack for my work based on NRI recently (old one is still opengl based and not keeping up with our needs), and I've started adopting NRI video encode/decode in it (our applications are heavily video focused).
So I'll actually be stressing this with real world workloads soon.

I also still plan to direct some attention at upgrading the Samples soon.

[Daedie-git]

I've reworked the NRISample. Now it generates a collection of patterns that are sensitive to encoding artifacts. It now also offers a small suite of quality controls that can be manipulated at runtime. And an option to show the diff between original and encoded->decoded image. This makes the sample actually a quite useful tool (I think my boss will want to have it :) ).

I have run into some bugs while integrating NRI video into our new graphics stacks. Those have been fixed.

I also added a script to the sample repo that does smoke runs of a rich matrix of configuration permutations, which I highly recommend keeping in some form. It was an essential tool for getting through this final stretch without accidentally re-introduce regressions.

I'm currently working through a final bug that surfaced in the aforementioned script. I expect to deal with it today or tomorrow. Then I actually think I'm ready for a final review + merging.

[Daedie-git]

I'm done for the most part.
The VIDEO_TODO.md has some remaining open ends. They relate to discrepancies between the backends for the most part. Dealing with them is a highly subjective matter and requires your input. They're probably worth addressing to get a really great developer experience, but I'm not quite sure how yet.

That being said, I would actually like to converge on a merge soon for this. Keep the above as documented known shortcoming. I think what's there is already quite strong even if not perfect.

[dzhdanNV]

Thanks. I updated this branch to the latest v180 release. Unfortunately, I will be on vacation until mid August and can't work on this right now. But I will check it out when I come back.

[Daedie-git]

Alright, have a great vacation!