Add ability to foveate a render target

[cabanier]

three.js is moving to a 2 (or more) pass solution.
This creates a problem for foveation since it's only applied to the textures of the swapchain. If we want to preserve foveation, we'd have to first render to the swapchain texture, then copy it to a regular one and then apply the second pass.

I propose that we add a new API to XRWebGLBinding to apply foveation to the current bound target. That way, the first render pass can render to that target and the next one can draw to the swapchain texture.

Proposed new API:
void foveateBoundTexture(GLenum target, float fixed_foveation)

/agenda Add ability to foveate a render target

[cabanier]

cc @AdaRoseCannon @zachernuk @toji to investigate if this approach will work on their XR systems.

[tangobravo]

Foveated rendering for arbitrary render targets seems like it should be a WebGL / WebGPU concern rather than a WebXR one to me?

[mwyrzykowski]

Foveated rendering for arbitrary render targets seems like it should be a WebGL / WebGPU concern rather than a WebXR one to me?

Agreed, seems preferable since there are use cases outside of XR environments.

[cabanier]

Foveated rendering for arbitrary render targets seems like it should be a WebGL / WebGPU concern rather than a WebXR one to me?

Agreed, seems preferable since there are use cases outside of XR environments.

I don't see where this could be used outside of a headset environment. Can you give an example?

[cabanier]

Foveated rendering for arbitrary render targets seems like it should be a WebGL / WebGPU concern rather than a WebXR one to me?

I agree it's a feature that is more suitable for WebGL. However, it's very complicated to set up and requires a lot of device specific information to set up correctly. WebXR already does this heavy lifting under the hood so it's easier to expose it there especially because it's not very useful in other situations.

[mwyrzykowski]

Foveated rendering for arbitrary render targets seems like it should be a WebGL / WebGPU concern rather than a WebXR one to me?

Agreed, seems preferable since there are use cases outside of XR environments.

I don't see where this could be used outside of a headset environment. Can you give an example?

Oh sure, it was actually brought up in gpuweb over five years ago: gpuweb/gpuweb#450

Something like 30+% improvement from VRR / foveation

[cabanier]

Foveated rendering for arbitrary render targets seems like it should be a WebGL / WebGPU concern rather than a WebXR one to me?

Agreed, seems preferable since there are use cases outside of XR environments.

I don't see where this could be used outside of a headset environment. Can you give an example?

Oh sure, it it actually brought up in gpuweb over five years ago: gpuweb/gpuweb#450

Something like 30+% improvement from VRR / foveation

What you describe there, is very similar to how it's done in OpenXR with Vulkan. Maybe we can make this a WebGPU extension? cc @toji

[toji]

As @mwyrzykowski pointed out, this has been brought up in the WebGPU group but it hasn't been a feature we've heard a lot of requests for so it's not on the immediate roadmap. We can certainly communicate that it's desirable for WebXR to the group, but I'd expect any implementation to be years out at this point. That said I agree that handling it at the graphics API level is probably the best long term solution.

A more limited foveation feature scoped just to WebXR layers still could be valuable in the meantime?

[cabanier]

A more limited foveation feature scoped just to WebXR layers still could be valuable in the meantime?

I think that's reasonable for WebXR layers (both WebGPU and WebGL) and I think my proposal for applying foveation to a WebGL texture would also work.
I'm unsure how this would would for regular Vulkan textures though.

[AdaRoseCannon]

/facetoface