XRGPUSubImage texture lifetimes

[toji]

An issue was raised during a code review for my WIP prototype of the proposal today: We should decide on and tightly specify the lifetime of the textures returned in an XRGPUSubImage.

I think the general assumption is that the textures will be supplied by a swap chain of some sort on the backend that cycles through a number of textures as needed to keep the graphics pipeline flowing. There's also scenarios, though, where supplying a single texture could be sufficient, such as if the developers asks for a depth/stencil texture but the system does not make use of that data for reprojection or occlusion.

We would want those details to stay opaque to the developer, though, so they don't try to rely on a given behavior that may not be present on other devices. As such we may want to mandate that even if a texture is reused on the backend a new JS wrapper is returned each frame to prevent developers from trying anything funny. (This, of course, comes with some overhead.)

A more complex concern also crops up that wasn't present with WebGL: Command buffers can be pre-recorded and executed at a later point. This means that You could theoretically retrieve the textures for a layer, record (but not submit) render commands with them, allow the frame to finish, and then submit the recorded commands while the textures are presumably already in-flight to the compositor.

WebGPU handles this scenario with the canvas by introducing a concept of "expiring the current texture" where the texture is marked as destroyed once the frame ends but the "drawing buffer" for that texture is retained for use by the compositor. I imagine we will probably need a similar concept for the WebGPU bindings or even link directly to the WebGPU spec language in that regard.

CC @mwyrzykowski, @cabanier

[mwyrzykowski]

Currently I believe the texture lifetime is at least that of the lifetime of a given XRGPUSubImage as they are annotated with [SameObject] right?

] interface XRGPUSubImage : XRSubImage {
    [SameObject] readonly attribute GPUTexture colorTexture;
    [SameObject] readonly attribute GPUTexture? depthStencilTexture;
    [SameObject] readonly attribute GPUTexture? motionVectorTexture;

So after the XRGPUSubImage's lifetime expires, then so could the GPUTexture instances. Though currently, we could return the same GPUTexture indefinitely for any number of new XRGPUSubImage instances.

Internally the GPUTexture can be a new JS object but backed by the same underlying MTLTexture, this is what we do for WebGPU today.

My assumption from the explainer is that xrGpuBinding.getViewSubImage is called per frame, which means the JS object lifetime of the texture expires after each frame similar to WebGPU.

We could also make that explicit and link to the WebGPU specification. That seems reasonable to me.

It seems like it would be even easier to specify in WebXR, as XRSession has requestAnimationFrame. So we could say that any GPUTexture instances returned from an XRGPUSubImage expire at the end of the requestAnimationFrame callback.

[cabanier]

I think the general assumption is that the textures will be supplied by a swap chain of some sort on the backend that cycles through a number of textures as needed to keep the graphics pipeline flowing. There's also scenarios, though, where supplying a single texture could be sufficient, such as if the developers asks for a depth/stencil texture but the system does not make use of that data for reprojection or occlusion.

I never like that we create textures that are not backed by swapchains just to be "consistent". Could we change WebGPU so the developer can decide ?
Currently, most implementations discard these anyway (if the UA indicates it doesn't need them) and substitutes them with their own which wastes resources.

We would want those details to stay opaque to the developer, though, so they don't try to rely on a given behavior that may not be present on other devices. As such we may want to mandate that even if a texture is reused on the backend a new JS wrapper is returned each frame to prevent developers from trying anything funny. (This, of course, comes with some overhead.)

The overhead is likely very minimal. Creating a new one each time sounds good to me.

A more complex concern also crops up that wasn't present with WebGL: Command buffers can be pre-recorded and executed at a later point. This means that you could theoretically retrieve the textures for a layer, record (but not submit) render commands with them, allow the frame to finish, and then submit the recorded commands while the textures are presumably already in-flight to the compositor.

WebGPU handles this scenario with the canvas by introducing a concept of "expiring the current texture" where the texture is marked as destroyed once the frame ends but the "drawing buffer" for that texture is retained for use by the compositor. I imagine we will probably need a similar concept for the WebGPU bindings or even link directly to the WebGPU spec language in that regard.

Awesome. This way we don't have to introduce a workaround like the "opaque texture" in WebGL.