[WebGPU-XR][Phase 3] WebXR projection matrix / NDC handling + upright projection-layer rendering - #18688
[WebGPU-XR][Phase 3] WebXR projection matrix / NDC handling + upright projection-layer rendering#18688RaananW wants to merge 4 commits into
Conversation
…engine RTT Y-flip) The WebGPU engine draws every render target Y-flipped (InternalsUBO yFactor = -1 at webgpuEngine.pure.ts, plus compensating front-face winding) so a later-sampled RTT matches the WebGL texture-space convention. XR projection-layer textures are presented directly by the XR compositor (top-left origin, per the WebXR/WebGPU spec's plain render pass) and are never re-sampled by Babylon, so that flip makes the in-headset image upside-down and its depth inconsistent with the [0,1] projection matrix the compositor uses for reprojection. Add an @internal WebGPURenderTargetWrapper._disableEngineYFlip flag, set only by the WebGPU XR render-target provider, that makes the engine render the target upright (non-inverting InternalsUBO + main-framebuffer winding), matching the canvas. Viewport/scissor stay in the top-left render-target branch since the XR sub-image viewport is already top-left. WebGPU + XR only: the flag is never set on the WebGL2 path or on non-XR targets, so WebGL2 XR stays byte-identical and non-XR WebGPU rendering is unchanged. Zero net public API (flag is @internal on the WebGPU subclass). Refs #18639 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Please make sure to label your PR with "bug", "new feature" or "breaking change" label(s). |
|
Snapshot stored with reference name: Test environment: To test a playground add it to the URL, for example: https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/18688/merge/index.html#WGZLGJ#4600 Links to test your changes to core in the published versions of the Babylon tools (does not contain changes you made to the tools themselves): https://playground.babylonjs.com/?snapshot=refs/pull/18688/merge To test the snapshot in the playground with a playground ID add it after the snapshot query string: https://playground.babylonjs.com/?snapshot=refs/pull/18688/merge#BCU1XR#0 If you made changes to the sandbox or playground in this PR, additional comments will be generated soon containing links to the dev versions of those tools. |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
🟢 Memory Leak Test Results4 passed, 0 leaked out of 4 scenarios 🟢 All memory leak tests passed — no leaks detected. Passed Scenarios (4)
|
|
Visualization tests for WebGPU |
⚡ Performance Test Results🟢 All performance tests passed — no regressions detected. |
|
WebGL2 visualization test reporter: |
…ebGPU) engines On a WebGPU engine (isNDCHalfZRange === true) the clip-space depth range is [0, 1], while WebGL/OpenGL uses [-1, 1]. WebXRCamera copies the XR binding's projection matrix verbatim and the rig cameras freeze their projection, so the engine's range-aware projection builders are bypassed. If the binding returns a [-1, 1]-convention matrix while the engine clips at [0, 1], every fragment with NDC z in [-1, 0) is clipped and all world geometry disappears (clear color still shows) - matching the in-headset "meshes flash a frame then vanish" symptom. Fix (WebGPU-gated, WebGL2 byte-identical, zero net public API): - Add @internal Matrix.convertProjectionToHalfZRangeInPlace(), reusing the engine's existing mtxConvertNDCToHalfZRange remap (the same conversion the perspective/orthographic builders apply when halfZRange is set). - In WebXRCamera._updateFromXRSession, on a half-Z engine, empirically detect the convention the binding used (project the view-space near point through the raw matrix: NDC z ~= -1 => [-1, 1], ~= 0 => [0, 1]) and convert only a [-1, 1] matrix, so a spec-compliant [0, 1] binding is never double-converted. Guards minZ <= 0 (avoids divide-by-zero) and assumes non-reverse-Z. WebGL2 (isNDCHalfZRange === false) never enters the block. Adds 6 unit tests: [-1,1]->[0,1] conversion, [0,1] left unchanged, WebGL2 byte-identical, asymmetric off-center frusta, minZ<=0 guard, and converter equivalence to the engine's half-Z builder. Refs #18639 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
🟢 Memory Leak Test Results4 passed, 0 leaked out of 4 scenarios 🟢 All memory leak tests passed — no leaks detected. Passed Scenarios (4)
|
|
Visualization tests for WebGPU |
|
WebGL2 visualization test reporter: |
⚡ Performance Test Results🟢 All performance tests passed — no regressions detected. |
🟢 Memory Leak Test Results4 passed, 0 leaked out of 4 scenarios 🟢 All memory leak tests passed — no leaks detected. Passed Scenarios (4)
|
|
WebGL2 visualization test reporter: |
|
Visualization tests for WebGPU |
⚡ Performance Test Results🟢 All performance tests passed — no regressions detected. |
🟢 Memory Leak Test Results4 passed, 0 leaked out of 4 scenarios 🟢 All memory leak tests passed — no leaks detected. Passed Scenarios (4)
|
|
WebGL2 visualization test reporter: |
ccd3a34 to
292e86f
Compare
|
Visualization tests for WebGPU |
🟢 Memory Leak Test Results4 passed, 0 leaked out of 4 scenarios 🟢 All memory leak tests passed — no leaks detected. Passed Scenarios (4)
|
|
WebGL2 visualization test reporter: |
|
Visualization tests for WebGPU |
⚡ Performance Test Results🟢 All performance tests passed — no regressions detected. |
🟢 Memory Leak Test Results4 passed, 0 leaked out of 4 scenarios 🟢 All memory leak tests passed — no leaks detected. Passed Scenarios (4)
|
|
Visualization tests for WebGPU |
|
WebGL2 visualization test reporter: |
⚡ Performance Test Results🟢 All performance tests passed — no regressions detected. |
Phase 3 (#18639, PR #18688) diagnostic — NOT a shipping change; reverted before Phase 3 lands. Discriminates whether a raw draw into the compositor sub-image presents when it is the SOLE writer of that sub-image, submitted on its OWN command encoder + OWN queue.submit (mirroring the working bare WebGPU-WebXR sample), run inside Babylon's XR frame loop. Confined to the WebGPU-XR provider (WebXRWebGPUCompositionLayer.ts); WebGL2 provider untouched. @internal, zero net public API. Distinct from the NDC and Y-flip commits. - Divert Babylon's per-eye scene render into a per-eye throwaway GPUTexture so the real compositor sub-image is never touched by Babylon this frame (no clear/draw), killing the endFrame single-submit overwrite confound. - Own-encoder + own-submit green-on-blue triangle into the real sub-image (createView only, never destroys the compositor texture). - Gated on a real WebGPU _device; NullEngine path is inert passthrough (exact repoint baseline), so the XR unit suite is unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
3f303ab to
ee019c9
Compare
🟢 Memory Leak Test Results4 passed, 0 leaked out of 4 scenarios 🟢 All memory leak tests passed — no leaks detected. Passed Scenarios (4)
|
|
Visualization tests for WebGPU |
|
WebGL2 visualization test reporter: |
⚡ Performance Test Results🟢 All performance tests passed — no regressions detected. |
|
@RaananW I opened #18711 for the separate capability-order prerequisite found during broader WebGPU runtime testing. #18711 contains only the minimal reorder and five browser/native, WebGPU/WebGL compatibility tests. It makes the existing WebGPU layer provider reachable; this draft can remain focused on projection/NDC and upright rendering once that path is active. There is no file overlap. |
ee019c9 to
8bee2b9
Compare
…rotating-pool fix) WebGPU composition-layer sub-image textures come from a small rotating pool that the compositor reads shortly after each eye is rendered. Babylon's default flow records both eyes into the single shared render encoder and submits it once, late, at endFrame; on the rotating pool that late submit can land after the slot has already been recycled, so the eye's contents never present (observed in-headset as the scene flashing for ~2 frames then going black). Fix: the WebGPU-XR composition-layer provider subscribes to onAfterCameraRenderObservable (fires per eye, after that eye is rendered into its sub-image target) and calls engine.flushFramebuffer() to submit that eye's recorded work immediately, before the compositor recycles the slot. This mirrors the reference WebXR/WebGPU sample's per-view submit timing. Confined to the WebGPU-XR provider; the WebGL2 XR path uses a different provider and is byte-identical. @internal, zero net public API. flushFramebuffer() is already used mid-frame elsewhere in the engine, so per-eye use is a supported pattern. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
8bee2b9 to
0c42466
Compare
|
Visualization tests for WebGPU |
|
WebGL2 visualization test reporter: |
🟢 Memory Leak Test Results4 passed, 0 leaked out of 4 scenarios 🟢 All memory leak tests passed — no leaks detected. Passed Scenarios (4)
|
⚡ Performance Test Results🟢 All performance tests passed — no regressions detected. |
|
Visualization tests for WebGPU |
|
WebGL2 visualization test reporter: |
## Background Broader WebGPU runtime testing showed that `WebXRLayers.isCompatible()` rejects every native session before checking the active engine and `XRGPUBinding` capabilities. A standards-compatible WebGPU runtime can therefore expose `XRGPUBinding.createProjectionLayer` and still be prevented from selecting the existing WebGPU projection-layer path. ## Change - Check WebGPU and `XRGPUBinding.createProjectionLayer` before the native WebGL fallback. - Keep native WebGL sessions on the existing `NativeXRRenderTarget` path. - Preserve browser WebGPU and WebGL behavior. - Make no public API changes. ## Tests Adds five explicit compatibility cases: - native WebGPU with projection-layer support - native WebGPU without `XRGPUBinding` - native WebGL with `XRWebGLBinding` - browser WebGPU with projection-layer support - browser WebGL with projection-layer support Local validation: - focused fixture: 5/5 tests passed - XR unit suite: 280/280 tests passed across 16 files - exact-file ESLint and Prettier checks passed - production core TypeScript build passed ## Relation to #18688 cc @RaananW. This is intentionally separate from #18688: this PR makes the WebGPU layer provider reachable when the runtime advertises the required capability, while #18688 addresses projection/NDC and upright rendering after that layer path is active. The changes do not overlap files.
3 participants
Phase 3 of WebGPU-for-WebXR (refs #18639)
DRAFT — pending in-headset validation on Meta Quest (WebGPU). Do not merge until RaananW confirms.
Builds on Phase 2 (#18655), which reached
IN_XRand rendered both eyes on Quest via the WebGPU projection-layer RTT provider but left two in-headset symptoms: the image was vertically flipped, and geometry disappeared (meshes flash for ~1 frame on entry, then are gone while the clear color remains).Root cause of the flip (proven)
The WebGPU engine draws every render target Y-flipped: it binds the inverting InternalsUBO (
yFactor = -1,webgpuEngine.pure.ts:3750) so the vertex shader doesposition.y *= yFactor_(webgpuShaderProcessorsWGSL.pure.ts:409), plus a compensating front-face winding flip (:3716). This exists so that a later-sampled RTT matches the WebGL texture-space convention.XR projection-layer textures are presented directly by the XR compositor (top-left origin, per the WebXR-WebGPU spec's plain render pass) and are never re-sampled by Babylon. So the engine's RTT Y-flip makes the in-headset image upside-down. This matches the confirmed in-headset report ("down is up, up is down").
The depth-range concern from the issue is confirmed benign: per spec, a WebGPU
XRSession'sXRView.projectionMatrixis authored for a[0,1]clip-space depth range, matchingWebGPUEngine.isNDCHalfZRange = true. It is consumed verbatim (webXRCamera.ts:323, andfreezeProjectionMatrixbypasses the engine's depth remap), so there is no double-conversion and no range clipping.On the disappearance — UNPROVEN hypothesis, with a known gap
This PR definitively fixes the flip. It may or may not fix the disappearance; that is an unproven hypothesis and the in-headset test is the sole decider — do not read this PR as a proven two-for-one.
The candidate mechanism: the same Y-flip flips the submitted depth relative to the
[0,1]projection matrix, and the compositor uses that depth. Known gap: the reported symptom is geometry gone while motionless, and with zero head motion positional reprojection/timewarp is ~identity, so flipped depth would not clearly warp geometry off-view without a warp delta. The depth-reprojection story therefore does not cleanly explain the motionless case. Treated as an open question, not a claim.Fix (WebGPU + XR only)
Add an
@internalWebGPURenderTargetWrapper._disableEngineYFlipflag, set only by the WebGPU XR render-target provider, that makes the engine render that target upright — non-inverting InternalsUBO (yFactor = +1) and main-framebuffer winding, exactly like the canvas. Viewport/scissor stay in the top-left render-target branch because the XR sub-image viewport is already top-left.webgpuRenderTargetWrapper.ts—@internal _disableEngineYFlip = falsewebXRWebGPURenderTargetTextureProvider.ts— set the flag on the created per-eye XR RTTwebgpuEngine.pure.ts— honor the flag at the two RTT Y-flip sites (yFactor bind + front-face winding); the derivedinvertYRenderTargetreproduces the original mapping exactly for canvas and non-XR RTTsConstraints upheld
RenderTargetWrapper, the WebGL provider, andwebXRCameraare untouched.@internalon the WebGPU subclass; side-effects manifest and tree-shaking barrels are unchanged.Gates (run locally, real deps)
tsc -b packages/dev/core/tsconfig.build.json→ exit 0vitest run --project=unit XR→ 275 passed (incl. a new test asserting the flag is set on the XR RTT and left false on plain RTTs)check:side-effects-sync→ up-to-dateprettier --checkclean;eslint0 errorsIn-headset read (Quest, WebGPU) — three branches
Refs #18639