Skip to content

Fix ntdll!RtlFreeHeap freeze: correct WM_INITDIALOG lParam extraction for property sheet pages - #93

Merged
katahiromz merged 2 commits into
masterfrom
copilot/fix-freeze-in-destroypropertysheetpage
Jul 2, 2026
Merged

Fix ntdll!RtlFreeHeap freeze: correct WM_INITDIALOG lParam extraction for property sheet pages#93
katahiromz merged 2 commits into
masterfrom
copilot/fix-freeze-in-destroypropertysheetpage

Conversation

Copilot AI commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

In MSYS2 builds, opening General Settings froze in ntdll!RtlFreeHeap because three dialog procs used as property sheet pfnDlgProc misinterpreted the WM_INITDIALOG lParam.

Root cause

When a dialog proc is used as a property sheet page's pfnDlgProc, Windows passes a PROPSHEETPAGEW* as lParam in WM_INITDIALOGnot the value stored in psp.lParam. The three affected procs cast lParam directly to their dialog class pointer:

// In XG_SettingsDialog::DialogProc, XG_RulePresetDialog::DialogProc, XG_HiddenDialog::DialogProc
s_pThis = (XG_SettingsDialog*)lParam;  // lParam is PROPSHEETPAGEW*, not &dialog1
SetWindowLongPtrW(hwnd, DWLP_USER, (LONG_PTR)lParam);

Subsequent member-variable writes (e.g. m_lfCellFont = xg_lfCellLogFont, m_bUpdating = TRUE) then stomped the comctl32-allocated PROPSHEETPAGEW heap block — overwriting dwSize, dwFlags, hInstance. When PropertySheetW tore down the pages, the corrupted block caused RtlFreeHeap to hang.

Fix

Added three thin wrapper procs in GUI.cpp (just before XgGeneralSettings) that intercept WM_INITDIALOG, replace lParam with the correct value from PROPSHEETPAGEW::lParam, then forward to the existing DialogProc:

static INT_PTR CALLBACK XgSettingsPageProc(HWND hwnd, UINT uMsg, WPARAM wParam, LPARAM lParam)
{
    if (uMsg == WM_INITDIALOG)
        lParam = reinterpret_cast<PROPSHEETPAGEW*>(lParam)->lParam;
    return XG_SettingsDialog::DialogProc(hwnd, uMsg, wParam, lParam);
}
// XgRulePresetPageProc and XgHiddenPageProc are identical in structure

XgGeneralSettings now sets psp.pfnDlgProc to these wrappers instead of the class DialogProc directly. The DoModal paths and the original DialogProc implementations are untouched.

…Param for property sheet pages

When a dialog proc is used as pfnDlgProc for a property sheet page, the
lParam in WM_INITDIALOG is a PROPSHEETPAGEW* (not the dialog object
pointer). The old code cast lParam directly to the dialog class pointer
(XG_SettingsDialog*, XG_RulePresetDialog*, XG_HiddenDialog*), causing
member-variable writes to corrupt the comctl32-allocated PROPSHEETPAGEW
heap block. This resulted in ntdll!RtlFreeHeap freezing in MSYS2.

Fix: add three thin wrapper dialog procs (XgSettingsPageProc,
XgRulePresetPageProc, XgHiddenPageProc) that intercept WM_INITDIALOG,
replace lParam with PROPSHEETPAGEW::lParam (the actual dialog object
pointer), and forward to the original DialogProc. Use these wrappers as
pfnDlgProc in XgGeneralSettings instead of the class DialogProc directly.

The existing DialogProc/DoModal paths are unchanged.
@katahiromz
katahiromz marked this pull request as ready for review July 2, 2026 01:58
Copilot AI changed the title [WIP] Fix freeze in ntdll!RtlFreeHeap during DestroyPropertySheetPage Fix ntdll!RtlFreeHeap freeze: correct WM_INITDIALOG lParam extraction for property sheet pages Jul 2, 2026
Copilot AI requested a review from katahiromz July 2, 2026 02:00
@katahiromz
katahiromz merged commit 7726948 into master Jul 2, 2026
1 check passed
@katahiromz
katahiromz deleted the copilot/fix-freeze-in-destroypropertysheetpage branch July 2, 2026 02:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants