Fix DWM fullscreen transition artifacts without affecting pre-Windows 8 behavior

[MarceloAlejandroJorquera]

Summary

This is a follow-up to the previous fullscreen/window-composition fix.

On Windows 8+ with DWM composition, rapidly entering/exiting fullscreen could still expose transient compositor states during the top-level window style/geometry transition. Depending on playback state, this could appear as:

• the desktop or windows behind MPC-BE becoming visible for a frame;
• an old/stale framed player image;
• the video renderer briefly appearing in the upper-left portion of the fullscreen surface;
• an intermediate snapshot size before the final fullscreen video geometry was established;
• similar transient artifacts with the empty player, audio-only playback, and audio cover-art/placeholder states.

The final implementation keeps the existing fullscreen logic intact and adds a short-lived, compositor-independent transition snapshot only on Windows 8 and later. The real MPC-BE window is temporarily cloaked while its final fullscreen/windowed style, geometry, child layout, and renderer rectangle are established. The snapshot remains in front until DWM has processed the final real-window state, then it is removed.

Root cause

The issue was not a normal repaint problem.

During a fullscreen transition, the main HWND changes non-client style, top-level rectangle, child layout, and video-renderer geometry in a very short interval. DWM can retain/present an older representation of the same HWND while those changes are being committed.

Simply hiding/showing the window, forcing synchronous repaint, or changing the order of SetWindowPos calls was not sufficient:

• forcing RedrawWindow during the hidden geometry transaction could publish stale saved client/non-client bits;
• cloaking without an independent visible surface could expose the desktop underneath;
• a normal popup cover was not reliable enough as a compositor quarantine surface;
• a full old-to-new screen capture could itself include background content;
• changing snapshot geometry after it had already been submitted caused visible intermediate resize states.

The working solution is therefore a compositor handoff rather than a repaint workaround.

Implementation

The transition path now:

1. Determines the transition surface and captures only the currently relevant MPC-BE content.
2. Builds an opaque black layered snapshot covering the transition area.
3. For active video, places the frozen frame directly at the destination geometry that MPC-BE's final video layout will use, avoiding a second/intermediate snapshot resize.
4. For empty/audio/static states, preserves the existing rendered content without synthetically scaling the whole old view.
5. Commits the independent layered snapshot to DWM before the real window is cloaked.
6. Cloaks the real MPC-BE HWND while keeping it logically visible so the normal MFC/layout path can run.
7. Applies the normal fullscreen/windowed style, top-level geometry, layout, and MoveVideoWindow() work.
8. Reasserts the snapshot above the real player before reveal.
9. For static/audio states, performs the final repaint only after the target geometry is installed and while the layered snapshot still hides the real HWND from the user.
10. Uncloaks the real player and waits for the required compositor synchronization while the snapshot remains in front.
11. Removes the snapshot only after the final real-window presentation is ready.

No playback pause/restart is introduced.

Responsiveness

The correctness-critical compositor ordering is intentionally unchanged after the visual fix was validated.

To reduce the cost of repeated fullscreen toggles, the backing GDI surface is cached:

• the compatible memory DC is reused;
• the 32-bit DIB is reused when the transition dimensions are unchanged;
• only the previously occupied content rectangle is cleared instead of zeroing the entire monitor-sized buffer on every toggle;
• the layered transition HWND itself is still created/destroyed for each transition, preserving the DWM lifetime/order that proved reliable.

This primarily improves rapid/repeated F toggling without weakening the synchronization that prevents the artifacts.

Windows compatibility / regression containment

This change is deliberately runtime-gated to Windows 8 and later.

The new snapshot/cloak path is entered only behind:

SysVersion::IsWin8orLater()

[Aleksoid1978]

No, I won't accept that. It's better to leave it as is or even roll it back to the way it was before.
There's no need for such complex code that also duplicates size calculations (and what if they change elsewhere in the future, etc.).

[MarceloAlejandroJorquera]

No, I won't accept that. It's better to leave it as is or even roll it back to the way it was before. There's no need for such complex code that also duplicates size calculations (and what if they change elsewhere in the future, etc.).

Understood. I can test a much smaller alternative, you'll judge if it too complex to maintain, the idea is to solve the last issue not addressed at first. Do you want me to keep trying or do you consider it done here, no further attempt needed?

[Aleksoid1978]

Yes - keep trying.
The best thing is to find a solution without using DWM_CLOAK.