Supported texture formats

[toji]

It was brought up that we will definitely want to limit the WebGPU texture formats which are accepted by the layer creation methods. For example: We probably don't want users requesting an r8sint color format with a stencil8 depth/stencil format.

WebGPU's solution for this regarding canvas integration is to limit the context to a narrow set of supported context formats. Specifically:

• bgra8unorm
• rgba8unorm
• rgba16float

(There is no limit on depthStencil formats because unlike WebGL the canvas does not supply depthStencil buffers. They are managed manually by the developer.)

We should determine how we want to limit WebXR's accepted texture formats. Either by putting a static list in the spec like WebGPU or by explicitly advertising the formats that may be used (though even in that case I would advocate for picking a format or two that are always guaranteed to be supported.)

Also worth considering is that WebGPU advertises a "preferred" format for the device which will avoid unnecessary copies. We will likely want the same, and the explainer already contains a getPreferredColorFormat() method. We should consider if that needs to be updated based on how we handle the supported format list, and whether or not a getPreferredDepthStencil() format is also necessary.

CC @mwyrzykowski, @cabanier

[mwyrzykowski]

Following the WebGPU approach seems reasonable. Specifically, supporting the viewFormats approach would be great for srgb, as the native format for some devices is bgra8unorm-srgb:

NOTE: Canvas configuration cannot use srgb formats like "bgra8unorm-srgb". Instead, use the non-srgb equivalent ("bgra8unorm"), specify the srgb format in the viewFormats, and use createView() to create a view with an srgb format.

[cabanier]

We should determine how we want to limit WebXR's accepted texture formats. Either by putting a static list in the spec like WebGPU or by explicitly advertising the formats that may be used (though even in that case I would advocate for picking a format or two that are always guaranteed to be supported.)

I already got a bug that Quest browser was unacceptably slow with its non-preferred texture format. I think we should only allow the preferred one since WebXR is so sensitive to performance.

[cabanier]

Following the WebGPU approach seems reasonable. Specifically, supporting the viewFormats approach would be great for srgb, as the native format for some devices is bgra8unorm-srgb

What device has srgb as its native format?
We've run into gamma issues before (Vision Pro is still broken here I believe) so we should make this very clear

(https://gpuweb.github.io/gpuweb/#dom-gputextureformat-bgra8unorm-srgb). Instead, use the non-srgb equivalent ("bgra8unorm"), specify the srgb format in the viewFormats, and use createView() to create a view with an srgb format.

I originally tried to use views to work around the gamma problem but they weren't consistently implemented so gamma was sometimes still applied :-\

[toji]

supporting the viewFormats approach would be great for srgb

That's a good point, and it makes me wonder if we need to start breaking the texture info out into mini descriptors so that we can get more specific about how each surface is used. ie:

gpuBinding.createProjectionLayer({
  color: {
    format: 'bgra8unorm',
    viewFormats: ['bgra8unorm-srgb'],
    usage: GPUTextureUsage.RENDER_ATTACHMENT | GPUTextureUsage.COPY_DST
  },
  depthStencil: { format: 'depth24plus' }
});

I think we should only allow the preferred one since WebXR is so sensitive to performance.

This is definitely a factor worth considering, though I think it would introduce a bit of dev pain. We'd want to report the format on the binding rather than wait for devs to find it out from the first frame's texture. Also, we should consider how HDR is supported eventually.

Out of curiosity, what is the preferred format for the Quest and AVP? I'm gonna guess Quest is rgba8-unorm because Android, and I suspect most desktops are bgra8-unorm. Not sure about the vision pro.

What device has srgb as its native format?

I don't think any devices consume sRGB formats, but all of them assume that the data in the texture is sRGB encoded anyway, which... sigh. Using an sRGB view of a non-sRGB format is an easy way for devs to get the right formatting of the values in their output without doing silly gamma conversions in their shaders.

[cabanier]

I think we should only allow the preferred one since WebXR is so sensitive to performance.

This is definitely a factor worth considering, though I think it would introduce a bit of dev pain. We'd want to report the format on the binding rather than wait for devs to find it out from the first frame's texture. Also, we should consider how HDR is supported eventually.

True

Out of curiosity, what is the preferred format for the Quest and AVP? I'm gonna guess Quest is rgba8-unorm because Android, and I suspect most desktops are bgra8-unorm. Not sure about the vision pro.

No, it's bgra8-unorm. Three.js already added a workaround since Quest browser is returned the wrong (hardcoded) value. See mrdoob/three.js#29221

[mwyrzykowski]

Apple Vision Pro uses either MTLPixelFormatBGRA8Unorm_sRGB or MTLPixelFormatRGBA16Float for the texture format with DisplayP3 for BGRA8 and ExtendedLinearDisplayP3 for RGBA16Float color space. These are the textures that come from the compositor's triple buffering mechanism.

So if this is limited to a 'preferred' format then we would need to allow multiple preferred formats, to support HDR as mentioned.

@cabanier the gamma correction should be fixed in https://bugs.webkit.org/show_bug.cgi?id=273603 but if you are still seeing it on visionOS 2.0 betas, certainly a bug report or simply a website where it reproduces would be great.

[toji]

Huh. I'm surprised that everyone's preferred LDR format is BGRA! It's tempting to say that devs either get that or an agreed upon HDR format (if supported) but I'm also pretty sure that we'll run into some Android-based devices (Pico or HTC maybe?) that prefer RGBA, since that's generally what the Android SurfaceFlinger requires.

[AdaRoseCannon]

/tpac

[toji]

After discussion at TPAC the tentative resolution is that we will attempt to support the same set of formats that WebGPU supports for canvas rendering (bgra8unorm, rgba8unorm, and rgba16float). If a device's runtime does not support the format natively a copy will be needed, but this is already the case for he base WebGPU API. The preferred LDR format for a device will be queryable, as is the case for WebGPU's canvas rendering.

HDR rendering is only partically resolved. We decided that rgba16float will be supported, but did not make any decisions on tone mapping, color spaces, etc. These will require more investigation, and possibly collaboration with the WebGPU WG, which has been working on solutions for these issues.

[cabanier]

I'd like to do some more investigation before we make this decision.
OpenXR returns a list of supported formats. If Quest doesn't support all these 3 formats natively, I don't think they should be supported by default as the extra copy will introduce too much overhead.

[toji]

Yes, agreed that we need more investigation.

The language in my previous comment is perhaps a little too strong. Where I said "tentative resolution" it would be more accurate to say "the direction the group wants to investigate, and report back on issues that we encounter." We have not concretely committed to supporting all of those formats in all cases, simply stated that it would be nice if we aligned with WebGPU in this regard and are going to attempt that during prototyping.

[mwyrzykowski]

Does Vulkan support texture swizzles on Quest similar to MTLTextureSwizzle?

I see some references in the Vulkan documentation to VkComponentSwizzle but not sure if this can be used in a way similar to Metal: if a backend requests RGBA for instance, we can create a texture view of the BGRA compositor texture with a swizzle mask which is almost free, avoiding the copy. I think it is only during pixel read backs from the canvas APIs where a copy is needed, which seems rare for the immersive session textures.

I didn't fully work through if there are any issues with that approach, but should be possible to avoid the copy if texture swizzle is supported.

[toji]

I don't think there's any way to specify to OpenXR the VkImageView that you want to use, which is where the component swizzle would be applied. Instead it takes SwapChains directly with arguments for the viewport and imageArrayIndex.

Vulkan's spec also says that you can't apply swizzles to framebuffer attachements:

The VkComponentMapping components member describes a remapping from components of the image to components of the vector returned by shader image instructions. This remapping must be the identity swizzle for storage image descriptors, input attachment descriptors, framebuffer attachments, and any VkImageView used with a combined image sampler that enables sampler Y′CBCR conversion.

So I don't think it could be applied at the WebGPU rendering side either, unfortunately. 😞