Handle games whose frame number freezes or restarts

[Remleo]

UEVR uses the frame number it reads from the game as the key into its pose pipeline, and that number is
not always monotonic: it freezes while some games sit in a fullscreen menu, and it restarts from 1
on a level load. Either way a pose gets read from a slot belonging to another frame, and the view comes
loose from your head and swims against it, faster the longer it lasts. That is the main problem here and
most of what follows is about it -- Problem 1, in FFakeStereoRenderingHook.cpp.

On top of it, a smaller one. On the VR_NativeStereoFix path UEVR keeps rendering the world into the
right eye after the game has stopped drawing it, while the left eye shows the black backdrop the game
switched to. The left eye is the correct picture of the two -- the right eye is the one that should have
stopped as well. Problem 2, in D3D12Component.cpp.

• Affects: any game whose frame number freezes or restarts
• Measured in: The Outer Worlds 2, Gylt, Silent Hill 2
• Size: 3 files, 66 lines of code
• Repro: profiles for The Outer Worlds 2 and Gylt, attached in the first comment

Apologies for the length. The code is small -- most of what follows is why it is shaped this way,
because the obvious version ("compare the number with the previous one") is the one that breaks, and
only measurements show it. If you would rather see the defects first, there are two videos in the first
comment.

---

Problem 1: poses read from the wrong slot

What goes wrong

The game's frame number is used directly as the pipeline key:

game frame number  ->  pipeline_states[number % QUEUE_SIZE]  ->  predictedDisplayTime  ->  xrLocateViews

OpenXR::on_pre_render_game_thread refreshes a slot in only two cases:

• the slot is empty, or
• the key jumped forward past it -- prev_frame_count + 1 < frame_count

Any other revisit keeps the predicted display time that is already in the slot. xrLocateViews then
predicts for a moment that has already passed, and the further behind it is, the faster the image
swims.

Both misbehaviours get there their own way, and there may well be more of them:

the game's number	what happens to the key	measured in
freezes while a fullscreen menu is up, though the game keeps rendering	every frame lands in one slot, occupied after the first	The Outer Worlds 2 -- held for 159 submitted view families
restarts from 1 on a level load	lands on slots still holding frames from before the load	all three; Gylt on every transition, including in and out of the main menu

Same visible result, same cause: a stale predicted display time reaching xrLocateViews.

The change

for each view family:

    number > highest seen        ->  real progress, remember the time
    number dropped by 64+        ->  restart:  carry += highest + 1 - number
    no progress for 250 ms       ->  stalled:  ++stall_frames
    stall just ended             ->  resumed:  carry += stall_frames, stall_frames = 0

    key = number + carry + stall_frames

Why it is shaped this way:

• An offset, not a counter of UEVR's own. A game that behaves keeps its own numbering, because
  the offset stays zero.
• The stall is folded into the carry, not dropped. Handing the key back at the end of a stall is
  another backwards jump -- and stalls here reached 159 frames.
• A restart lands exactly one above the highest. No gap, no overlap.
• 64 tells a restart from extra view families. A restart drops the number by orders of magnitude;
  extra families differ by a few.
• Both publishers add the same offset. begin_render_viewfamily and the render thread path both
  name a frame to the runtime. Apply it in one and the two disagree by the width of a stall, so a
  pose stored under one key is looked up under another. The render thread path is only spared this by
  returning early when VR_NativeStereoFix is on -- which is not a guarantee.

Why not number == previous number?

Because a game can push several view families per frame, and then that test breaks in both
directions:

the game submits	== test	consequence
a repeated number within a frame	passes during ordinary play	key shifts a frame every frame, view answers head movement with amplified jitter
alternating numbers	almost never passes	a real stall goes unnoticed

Measured, not reasoned:

• Gylt and Silent Hill 2 both submit a repeated number within a frame -- and both jittered with an
  equality test in place
• The Outer Worlds 2 submits one number per call and showed nothing at all, with identical settings

Which pattern a game shows is up to the game, so a single title proves nothing either way.
Exceeding the highest number seen is immune to both.

---

Deciding that the game has stalled

Both fixes rest on this one question and pass nothing else between them: is_game_frame_stalled()
in VR.hpp, answered from the timestamp of the last real advance of the number.

• Answered at the point of use, not precomputed where the number is published. A game that stops
  running that path cannot leave a stored flag stuck at its last value.
• 250 ms, measured in time. "Produced no frame" and "produces frames slower than we present"
  differ only in duration. A count of presents would depend on the game's frame rate against the
  headset's; a count of calls, on how many view families the game pushes.

---

Problem 2: the right eye renders a world the game has stopped drawing

On the VR_NativeStereoFix path only, and only while the number is frozen. It calls the detector above
and nothing else.

What goes wrong

VR_NativeStereoFix builds the stereo pair from two different sources:

left eye   <-  the game's own backbuffer
right eye  <-  a scene capture UEVR drives itself

While a fullscreen menu is up, the game stops drawing the world into its backbuffer. That is deliberate
-- the menu is meant to sit on a black backdrop. UEVR's scene capture does not know that and keeps
rendering the world.

left eye   the menu, on black   <- correct, that is what the game is presenting
right eye  the world, frozen    <- wrong, the capture kept going

Nothing is wrong with the left eye: it is showing what the game presents. The fault is on the right --
it holds a still picture of a world the game has stopped drawing, and with one eye holding a scene and
the other holding none of it, turning your head reads as the image rotating the wrong way.

The change

While the game is stalled, take the right eye from the backbuffer as well:

right_source = is_game_frame_stalled() ? backbuffer : scene capture

Both eyes then show what the game is actually presenting -- the menu, on the backdrop the game intended
-- and the capture comes back on the frame the game resumes. The backbuffer is already what the existing
m_scene_capture_tex == nullptr branch feeds the right eye, so this routes into behaviour that is there
already.

---

Why one PR and not two

Two defects, two places, but they hang off one shared piece:

        highest number seen  +  a 250 ms timer          <- the detector
                        |
          +-------------+-------------+
          |                           |
   Problem 1: the key           Problem 2: eye source
   FFakeStereoRenderingHook     D3D12Component

• The two consumers never touch each other. The key never looks at the eye source, or the other
  way round.
• Both need the detector, so splitting does not give you two independent PRs -- it gives you two
  sequential ones, with the detector travelling in whichever lands first.
• One piece would stand alone: the restart carry needs only "the highest number seen", not the
  timer.

The detector is also the part worth arguing about -- progress by the highest number rather than the
previous one, decided at the point of use, on a time threshold. Together it gets reviewed once.

Happy to split it either way if you would rather -- say which and I will.

---

Logging

Neither the stall nor the pattern is visible any other way: the game keeps rendering, so nothing else
in the log changes when its number freezes.

Game stopped advancing its frame number, holding N
Game resumed advancing its frame number at N, after M frames of stall, key at K
Game restarted its frame numbering (N after a high of M), carrying the key on to K
Game submits repeated frame numbers within a frame (saw N M times in a row)
Game alternates frame numbers per frame (N after M)

• Stall edges carry the key, so a log on its own shows whether it ever went backwards.
• The pattern lines fire once each.
• A repeat only counts while the run stays short. A stalled game repeats its number on every call, so
  an unbounded test would report every stall as the very pattern it has to be told apart from:
  several view families make a run of several, a stall makes a run of hundreds.

---

Reproducing it

Both profiles I used are attached in the first comment.

The freeze shows both problems at once. The Outer Worlds 2, with its profile -- it is what makes
the menu usable in VR:

1. Hold Back for about a second to open the pause menu
2. Switch to the Inventory or Companions tab
3. Turn your head from side to side

The restart shows Problem 1. Gylt, with its profile. Inject while the game sits in its
main menu, not after loading a save: UEVR has to be hooked before the level load, otherwise it never
sees the number restart and nothing goes wrong.

1. Inject in the main menu
2. Press Continue to load a save
3. Look around as the level comes up -- the image comes loose from your head and swims against it

The Outer Worlds 2 profile only sets mod values that are already in master -- VR_AimMethod,
VR_RoomscaleMovement, VR_CameraForwardOffset/UpOffset, UI_Distance/UI_Size -- and calls
recenter_view(). It has UObjectHook_EnabledAtStartup=false and its scripts never touch
UObjectHook, so it does not depend on anything unmerged.

Rolling your own profile instead? Then turn VR_NativeStereoFix on, or there is nothing to see:
with it off neither game shows the defect at all. Both of my profiles also have
VR_NativeStereoFixSamePass on, and that pair is what I measured. I did not try VR_NativeStereoFix on
with VR_NativeStereoFixSamePass off, so I cannot say whether that half matters here. The rendering
method was VR_RenderingMethod=0 in both.

---

Does this depend on VR_NativeStereoFix?

Yes. Measured both ways on plain master, with VR_NativeStereoFix the only thing changed:

game	steps	VR_NativeStereoFix on	off
Gylt	inject in the main menu, then Continue	swims	nothing
The Outer Worlds 2	open the pause menu, turn your head	swims	nothing

For Problem 2 that follows from the code: that copy only exists on the VR_NativeStereoFix path.

For Problem 1 it does not, so why VR_NativeStereoFix decides whether the swim shows up is worth a note.
From reading the code, the render thread publisher bails out on a repeated number only when it is on:

on    render thread returns early  ->  the slot keeps its stale display time  ->  swim
off   render thread keeps going    ->  enqueue_render_poses and synchronize_frame write
                                       pipeline_states[...] unconditionally, overwriting it

That part is a reading, not a measurement -- I did not instrument it.

What it means for risk: with VR_NativeStereoFix off, this change is inert in every game I could
test. The offset is still applied by both publishers, so the two cannot name a frame differently.

That is not a niche configuration, though. Without VR_NativeStereoFix plus
VR_NativeStereoFixSamePass, The Outer Worlds 2 renders shadows in one eye only: the game ships no
Instanced Stereo permutations, so the second eye arrives as eSSP_SECONDARY and parts of the renderer
skip it on IStereoRendering::IsASecondaryView(). VR_NativeStereoFixSamePass is what flips that
enum back. All three games I tested have both on, and this one needs them. I would expect the same of
most modern UE titles -- that part is an expectation, not something I measured.

---

Scope

Problem 2's fix touches the D3D12 + OpenXR path only. The same scene capture feeds the right
eye in three more places:

• D3D12 + OpenVR
• D3D11 + OpenXR
• D3D11 + OpenVR

Each sits next to a branch that already copies from the backbuffer, so the change would be the same
shape in all four. I can only test one: I have no D3D11 game that stalls, and The Outer Worlds 2 is
not playable on OpenVR at all for unrelated reasons. Happy to extend it if you would rather have all
four in one go, but three of them would be untested.

---

Tested

Release build, Quest 3 over Virtual Desktop, OpenXR, D3D12.

Every transition was checked against the log. The key lands exactly one above the highest value
it had used -- no gap, no overlap:

game	case	log	key
The Outer Worlds 2	menu, three times	stalls of 73, 113, 62 frames	21519, 21744, 21908
The Outer Worlds 2	menu, VR_NativeStereoFix off	stalls of 106, 75 frames	1231, 1877
Gylt	main menu and back, injected after loading	restart from 6437, then 110	6438, 6548
Gylt	same, injected in the main menu	restart from 19161	19162
Silent Hill 2	level load	restart from 4349	4350

The Gylt row injected in the main menu is the case that used to break: it swam on master, and still
swam with only the freeze handled.

The VR_NativeStereoFix off row is a consistency check, not a regression -- nothing visibly broke there
before or after. It is in the table because that is the configuration where the two publishers would
name a frame differently if the offset were applied in only one of them.

Sample lines:

14:07:15.815  Game stopped advancing its frame number, holding 1124
14:07:18.764  Game resumed advancing its frame number at 1125, after 106 frames of stall, key at 1231
13:58:40.903  Game restarted its frame numbering (1 after a high of 19161), carrying the key on to 19162

Ordinary gameplay is untouched. With a game that neither freezes nor restarts, the offset stays
zero and the key is the game's own number.

[Remleo]

Attaching the profiles I promised, so both defects can be reproduced without building one:

• The Outer Worlds 2 -- for the freeze. Hold Back for about a second to open the pause menu,
  switch to the Inventory or Companions tab, and turn your head. TheOuterWorlds2-WinGDK-Shipping.zip

• Gylt -- for the restart. Inject in the main menu, then press Continue. Gylt-Win64-Shipping.zip

VR_NativeStereoFix and VR_NativeStereoFixSamePass are on in both -- with them off neither defect
shows up at all.

Two videos, both The Outer Worlds 2 on plain master (4ee5c6b), no patch, VR_NativeStereoFix on. The
pause menu is open and I am turning my head:

• Left eye -- Problem 1 on its own. The image comes loose from my head and swims against it. The
  backdrop behind the menu is black, which is what the game presents and what both eyes should show.

animation.gif.mp4

• Right eye -- both problems at once. The same swim, plus the world still being rendered behind the
  menu, where the left eye has black.

53ce1ba80495015969b8e9be892ad363.2.mp4