Running Msvc Asan against win32 game

[xezon]

I found it particularly difficult to run the i386 Msvc Asan against the win32 version of the game.

Msvc Asan refers to the clang_rt.asan_dynamic-i386.dll which can be found in the Visual Studio installation folder and is enabled with the /fsanitize-address compile option.

https://learn.microsoft.com/en-us/cpp/build/reference/fsanitize?view=msvc-170

Problems:

Asan shadow memory collision

Asan will fail to allocate memory in a preferred region, because binw32.dll claims a portion of that range first:

==19316==Shadow memory range interleaves with an existing memory mapping. ASan cannot proceed correctly. ABORTING.
This can happen for multiple possible reasons:
   1. There is a module in the range of the shadow memory
   2. Some binaries are compiled with the Static CRT /MT(d) and some are built
      with the Dynamic CRT /MD(d)
       - this needs to be consistent across all binaries in a process, or ASan may
         try to initialize the runtime twice
   3. The function stack may be in the range of the shadow memory. This can
      sometimes be worked around by restarting the computer and trying again
==19316==ASan shadow was supposed to be located in the [0x2fff0000-0x3fffffff] range.
==19316==Dumping process modules:
    0x00240000-0x01433000 C:\Generals\English\Command & Conquer Generals Zero Hour\generals.exe
    0x01920000-0x01923000 C:\WINDOWS\SYSTEM32\sfc.dll
    0x21100000-0x2115f000 C:\Generals\English\Command & Conquer Generals Zero Hour\mss32.dll
    0x30000000-0x3006d000 C:\Generals\English\Command & Conquer Generals Zero Hour\binkw32.dll
    0x529f0000-0x529fb000 C:\Generals\English\Command & Conquer Generals Zero Hour\AVIFIL32.dll
    0x52a00000-0x52a37000 C:\WINDOWS\SYSTEM32\DINPUT8.dll
    0x52a40000-0x52a4e000 C:\WINDOWS\SYSTEM32\SortServer2003Compat.dll
...

ASan shadow was supposed to be located in the [0x2fff0000-0x3fffffff] range.
collides with
0x30000000-0x3006d000 C:\Generals\English\Command & Conquer Generals Zero Hour\binkw32.dll

It was not possible to set a different Asan memory region with environment variable ASAN_SHADOW_BASE. I worked around this issue by rebasing the address of binkw32.dll to another address:

"C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\bin\Hostx64\x86\editbin.exe" /REBASE:BASE=0x40000000 BINKW32.DLL

Fault tolerant heap collision

Windows will enable the Fault Tolerant Heap (FTH) when enabling compatibility for an executable.

This FTH collides with Asan and causes it to crash on boot.

I removed the compatibility mode to get rid of this problem.

I did it through regedit, locating the relevant key in Computer\HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers and deleting it. I expect disabling compatibility in the properties of generals.exe would also work.

Disabling the FTH via registry key Computer\HKEY_LOCAL_MACHINE\Software\Microsoft\FTH\Enabled did not work.

Allocating a 4 byte aligned std::string crashes on boot

std::string is 4 byte aligned in 32 bit arch. This somehow trips Msvc Asan, as it is expects an 8 byte alignment. Asan places a call in std::string functions to verify this alignment and then errors when violated.

Upgrading to latest msvc 2022 update and rechecking...

Edit: After upgrading VS2022 and grabbing the new clang_rt.asan_dynamic-i386.dll, it works correctly!

[xezon]

Asan for win32 works ok, but only for simple tests. After just 2 or so level loads Asan will run out of memory and crash. So for more in-depth checks we certainly need to upgrade to x64.

[Mauller]

"C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.43.34808\bin\Hostx64\x86\editbin.exe" /REBASE:BASE=0x40000000 BINKW32.DLL

For most recent VS22

[Mauller]

I found that to get ASAN working for generals, you cannot rebase the address on the provided BINKW32.dll as it has been signed, but copying the rebased ZH version works fine.

[Caball009]

Some comments that might help the next person:

1. There's an option in CMake to enable asan (RTS_BUILD_OPTION_ASAN).
2. RTS_GAMEMEMORY_ENABLE needs to be disabled in CMake to even start the game (and get beyond the splash screen) without running into issues found by asan.
3. In case anyone encounters an error ("sanitizer_win.cpp not found") on game launch, this is fixed by rebasing binkw32.dll as described above.