GPU: OpenXR integration

[Beyley]

After taking a deep dive into the various graphics API init routines for OpenXR (and reading the entire core specification front to back), I think the only way to make this work cleanly and according to the OpenXR spec is if SDL fully wrapped the creation of your OpenXR Instance, System, and Session into the same function call that creates the SDL_GPUDevice*. There is just way too much intertwined with the core init code of the GPU device, since all the extensions tell you which GPU device/instance/adapter you must use for rendering with OpenXR.

Along with this, there should likely be an SDL_GPUCreateXrSwapchain function, which picks the best format to use, and returns the created swapchain, and also SDL_GPUEnumerateXrSwapchainImages, which returns the SDL_GPUTexture* for every swapchain image in the swapchain you need to render to.
The SDL_GPUCreateXrSwapchain function can probably be omitted if desired, by having an SDL_GPUGetXrSwapchainFormats, which internally calls xrEnumerateSwapchainFormats and returns all the ones SDL_GPU can render to.

Init steps for each API

These are written from the perspective of SDL creating a GPU device from scratch (after it has determined which API it wants to use for VR)

XR_KHR_D3D11_enable

• Create an XrInstance with the XR_KHR_D3D11_enable extension enabled.
• Create the XrSystem
• Call xrGetD3D11GraphicsRequirementsKHR with the created OpenXR instance and system to get the feature level and adapter to use when creating your D3D11 objects.
• Create your D3D11 device and such
• Create an XrSession with the next ptr of XrSessionCreateInfo being a XrGraphicsBindingD3D11KHR with the created ID3D11Device
• Continue creating the SDL_GPUDevice* as normal

XR_KHR_D3D12_enable

• Create an XrInstance with the XR_KHR_D3D12_enable extension enabled.
• Create the XrSystem
• Call xrGetD3D12GraphicsRequirementsKHR to get the minimum feature level and adapter to use.
• Create your D3D12 device and a command queue
• Call xrCreateSession with XrSessionCreateInfo.next pointing to a XrGraphicsBindingD3D12KHR with the created ID3D12Device and ID3D12CommandQueue
• Continue creating the SDL_GPUDevice* as normal

XR_KHR_metal_enable

• Create an XrInstance with the XR_KHR_metal_enable extension enabled.
• Create the XrSystem
• Call xrGetMetalGraphicsRequirementsKHR to get the MTLDevice to use
• Create the GPU objects, including a command queue
• Create an XrSession with the next ptr of XrSessionCreateInfo pointing to XrGraphicsBindingMetalKHR filled in with the created command queue.
• Continue creating the SDL_GPUDevice* as normal

Note about Vulkan

From what I can see, you probably need to support both the XR_KHR_vulkan_enable and XR_KHR_vulkan_enable2 extensions, as some runtimes may only support one or the other.

XR_KHR_vulkan_enable

• Create an XrInstance with the XR_KHR_vulkan_enable extension enabled
• Create the XrSystem
• Call xrGetVulkanGraphicsRequirementsKHR to get the minimum/maximum Vulkan version supported by the OpenXR runtime
• Call xrGetVulkanInstanceExtensionsKHR to get the instance extensions to enable when creating your VkInstance
• Create your VkInstance
• Call xrGetVulkanDeviceExtensionsKHR to get the Vulkan device extensions to enable when creating your VkDevice
• Call xrGetVulkanGraphicsDeviceKHR to get the VkPhysicalDevice to use
• Create your VkDevice, pick a queue family+index
• Create an XrSession with the XrSessionCreateInfo.next ptr pointing to a XrGraphicsBindingVulkanKHR with the VkInstance, VkPhysicalDevice, VkDevice, queue family, and queue index to use.
• Continue creating the SDL_GPUDevice* as normal

XR_KHR_vulkan_enable2

• Create an XrInstance with the XR_KHR_vulkan_enable2 extension enabled.
• Create the OpenXR System
• Call xrGetVulkanGraphicsRequirements2KHR to get the minimum/maximum Vulkan version supported by the OpenXR runtime
• Call xrCreateVulkanInstanceKHR to create your VkInstance
• Call xrGetVulkanGraphicsDevice2KHR to get the VkPhysicalDevice to use
• Call xrCreateVulkanDeviceKHR to create the VkDevice to use
• Select your queue family/queue index
• Create the XrSession with the XrSessionCreateInfo.next ptr pointing to a XrGraphicsBindingVulkanKHR with the VkInstance, VkPhysicalDevice, VkDevice, queue family, and queue index
• Continue creating the SDL_GPUDevice* as normal

Brief summary of the handles that need to be passed between OpenXR and SDL, mostly to give scale of just how intertwined the init of the GPU device and OpenXR session really are.

List of GPU handles needed to pass into OpenXR

• ID3D11Device*
• ID3D12Device*
• ID3D12CommandQueue* (NOTE: I haven't used OpenXR with D3D12, but I from reading the spec, I believe this can be any D3D12 command queue)
• MTLCommandQueue (it actually wants an "an Objective-C object that conforms to the MTLCommandQueue protocol", but I dont know enough metal/obj-c to know what that entails)

XR_KHR_vulkan_enable

• VkInstance
• VkPhysicalDevice
• VkDevice
• Vulkan queue family index to use
• Vulkan queue index to use

XR_KHR_vulkan_enable2

• PFN_vkGetInstanceProcAddr
• const VkInstanceCreateInfo*
• const VkAllocationCallbacks*
• VkPhysicalDevice
• const VkDeviceCreateInfo*
• const VkAllocationCallbacks*

List of GPU handles returned by OpenXR, needed to be passed into the GPU API

• ID3D11Texture2D*, returned by xrEnumerateSwapchainImages
• ID3D12Resource*, returned by xrEnumerateSwapchainImages
• MTLTexture, returned by xrEnumerateSwapchainImages. (once again it's actually a void*, and is defined as "texture is populated with a valid Metal texture to use, which must be able to be bridged casted to an Objective-C object that conforms to the MTLTexture protocol")
• MTLDevice, returned by xrGetMetalGraphicsRequirementsKHR. The spec seems to force you to use a specific MTLDevice on metal, you can't create your own.
• VkImage returned by xrEnumerateSwapchainImages
• LUID for the D3D11/D3D12 adapter to use

XR_KHR_vulkan_enable

• VkPhysicalDevice, like Metal, it seems that Vulkan with OpenXR also forces you to use a particular VkPhysicalDevice, and the application is unable to pick one on it's own.

XR_KHR_vulkan_enable2

• VkInstance
• VkPhysicalDevice
• VkDevice

The various OpenXR extensions also return some required info to init a functional GPU device for OpenXR usage

• Minimum feature level for D3D11 device, returned by xrGetD3D11GraphicsRequirementsKHR.
• Minimum feature level for D3D12 device, returned by xrGetD3D12GraphicsRequirementsKHR
• (XR_KHR_vulkan_enable) Minimum/Maximum Vulkan version supported by the OpenXR runtime, returned by xrGetVulkanGraphicsRequirementsKHR
• (XR_KHR_vulkan_enable2) Minimum/Maximum Vulkan version supported, returned by xrGetVulkanGraphicsRequirements2KHR

[flibitijibibo]

Agreed - in the nuclear case we can just add a property but it'd be nice to have a full list of what apps need to support VR with any given GPU API.

We could maybe kill two birds with one stone by having an OpenXR sample at https://github.com/TheSpydog/SDL_gpu_examples, and developing a gpu-xr PR to support it. My only experience with XR right now is input, for #4464, so I dunno how much help I'll be.

[Beyley]

it'd be nice to have a full list of what apps need to support VR with any given GPU API.

I've updated the issue with a full breakdown of the init process for every currently supported graphics API, turns out this will probably be a lot more involved than I originally thought. But should still be possible without exposing any raw GPU handles to the end user.

We could maybe kill two birds with one stone by having an OpenXR sample at TheSpydog/SDL_gpu_examples, and developing a gpu-xr PR to support it. My only experience with XR right now is input, for #4464, so I dunno how much help I'll be.

I might fork off SDL and SDL_gpu_examples at some point, to try to actually get some code down for this, unless someone gets to it before me

[Beyley]

I have started very early work on this here and here. Right now I have instance/system/session creation working on Vulkan with KHR_vulkan_enable2. (side note: my SDL is saying its forked off that other repo since i've forked SDL in the past for other stuff)

I have nothing visual to show yet, since I still gotta write the rest of the surrounding code in the example itself, and also write some swapchain related functions in SDL_gpu to let me render to the XR session swapchain.

Only after that can I actually start working on cleaning this up to upstream. (which includes stuff like making the OpenXR loader dynamically loaded)

I'm terrible with CMake, so forgive my dirty hacks and probable sins in the CMake patches to SDL 🥲

[corentin35000]

Is this still relevant today? I would be very interested in developing VR games next!

[Beyley]

Is this still relevant today? I would be very interested in developing VR games next!

I haven't worked on my fork in about a month or so, but I'm still interested in making this happen, I'm just very busy with other projects at the moment, feel free to contribute to my fork if you want to see this make progress. Right now it desperately needs a rebase

[Beyley]

After some hacking today, i managed to get us all the way to rendering to the swapchain (with some hacks/hardcoded values)

big problem is that Monado's debug view is showing massive texture corruption (however the main simulated HMD's view is displaying correctly!)
im assuming this is some UB im hitting with vulkan, but given the app crashes in renderdoc (and the fact I don't really know much vulkan), I'm not entirely sure yet how to debug it, but i've at least made some progress!

[Beyley]

another update: i got it all working!
top left is a "desktop view of the scene", rendered myself to an SDL window using SDL_gpu
middle is Monado's debug interface, showing the composited image raw
bottom right is the GUI used for controlling the QWERTY virtual HMD, showing a readback from the GPU of the composited image (should always match the debug interface's view)

not drawing anything pretty, but it's all working, aside from some vulkan validation issues on app close im not entirely sure how to solve yet.