Guarantees about layer texture shape

[toji]

Right now the explainer indicates that for any given layer you request a sub image for each view/eye and it returns a combination of a GPUTexture and a view descriptor. This implicitly allows a very wide range of texture shapes to be returned:

• An array texture with a layer per view.
• A single texture with separate viewports for each view (XRWebGLLayer-style).
• A separate texture per view.
• A 3D texture with a depth slice per view.
• Some absolutely insane combination of the above.

While the API shape accommodates any of that , it would be a good idea (in my opinion) to limit the possible texture shapes that can be returned from the API so that developers have some guarantees about the patterns that can be used safely.

For example: If each view corresponds with a separate texture or layer you can safely clear the texture with every render pass. However if each view corresponds to a viewport into a single larger texture then you have to make sure you only clear the texture on the first render pass (WebGPU has no built-in way to clear only a subsection of a render attachment, so you would end up overwriting the first view's rendering on the second pass.) On the other hand, if using the multiple-viewport pattern some developers would be inclined to try and render both views in a single render pass. That would fail if some browsers returned a single texture and others returned an array texture.

Furthermore, some techniques such as multiview rendering rely on the render target being an array texture. While the technique is not supported in WebGPU today if we want to plan for it in the future we would either need to mandate array texture usage today or introduce a configuration option in the future that allows developers to require that a layer is backed by an array texture.

I seem to recall discussing this in the past with @cabanier and I believe that at the time we were both in favor of requiring WebGPU textures to always be array textures (when multiple views are necessary, anyway). Still, it would be good to confirm that and get the decision recorded in the repo. 😁

(One last consideration: If we DID constrain the API to only return array textures we might want to take a moment to re-evaluate the API shape because then there would be no specific reason to return the texture on a per-sub-image basis.)

CC @mwyrzykowski

[Kangz]

Are we confident that native XR presentation APIs now and in the future will use texture arrays? If they start using separate textures per view, the fact the WebXR requires a texture array to be used means that we'll have to emulate a 2D-array textures on top of a set of single-layer textures in the WebGPU implementation, which would be quite tricky to do (or impossible for sampling as texture_2d_array<f32>).

Also aren't there cases where the views have different sizes (for foveated rendering or other magics), such that we might need multiple different arrays texture with different width and height?

+1 on being more strict as to what can be returned, but overall it seems we should allow per-view array texture + layer index.

[cabanier]

+1 on only returning texture arrays.
They work better on mobile hardware since developers sometimes sometimes want to do a render pass just before each eye and this is very slow with a side by side texture. They are also required for multiview.

If there's a need for side-by-side, we can always add it later.

[cabanier]

Are we confident that native XR presentation APIs now and in the future will use texture arrays?

It's possible that different textures will come up in the future, but texture arrays will always be supported. I believe pretty much every native application on Quest uses them.

Also aren't there cases where the views have different sizes (for foveated rendering or other magics), such that we might need multiple different arrays texture with different width and height?

Quest has a fairly new vulkan-only rendering technique where each texture is divided into regions. AFAIK that still uses texture arrays

[mwyrzykowski]

Could we limit the shape to these two initially?

• An array texture with a layer per view.
• A separate texture per view.

Or do we have platforms where it is expensive to create a view of a texture from a texture array? This is supported by WebGPU, so I'm guessing the answer is no, but @cabanier or @toji know? On Metal it is certainly fine.

[cabanier]

Could we limit the shape to these two initially?

• An array texture with a layer per view.
• A separate texture per view.

Or do we have platforms where it is expensive to create a view of a texture from a texture array? This is supported by WebGPU, so I'm guessing the answer is no, but @cabanier or @toji know? On Metal it is certainly fine.

We currently use texture views on the GL side and they have no overhead so it's very likely the same for vulkan.
Given this, just an array texture should be fine.

[toji]

I'm not aware of any platforms where views of array textures are particularly expensive. (Possibly WebGPU's OpenGL-based compat mode in Chrome? But I think it's an open question whether or not we'll be supporting that with WebXR.)

I'd be fine with limiting it to those two if we felt that just array textures was too limiting. At the very least it means that the clear behavior is the same between either path. But I'd want to know that there was a scenario in which separate textures was preferred first? Does Vision OS have a preference for how these surfaces are supplied?

(There IS a reason why Chrome would want it. We don't yet support array textures for our cross-process texture mailboxes, so if that's required by the API then we'll need to do a copy while we sort that out. I've already done that code, however, and I don't want to introduce API restrictions for what should be a short-term, solvable problem on the part of Chrome.)

[mwyrzykowski]

@toji visionOS currently returns a single MTLTexture with type=MTLTextureType2DArray, arrayLength=2

[mwyrzykowski]

It is trivial to support separate textures on Metal with -[MTLTexture newTextureViewWithPixelFormat:textureType:levels:slices:], in fact I think we need to use that in any case since the texture type is MTLPixelFormatBGRA8Unorm_sRGB (or MTLPixelFormatRGBA16Float, but in that case we don't need the view)

[toji]

Great! In that case I would say let's aim for returning an array everywhere, and report back if for some reason that's not going to be practical in any known implementation.

I think we probably can extend it to returning separate textures in the future if we find a need. The worst-case scenario would be introducing side-by-side textures again, something I don't think there's any appetite for.