[CI] SOS.UnitTests desktop.cli tests fail with 'Failed to load data access module' (mscordacwks.dll mismatch)

[steveisok]

Summary

15 SOS.UnitTests with config: desktop.cli (and 2 DualRuntimes tests) are consistently failing on all Windows rolling CI legs (diagnostics-public-ci, definition 25) since approximately Mar 10, 2026. The pipeline is currently red — the last 2 consecutive rolling builds both failed.

Root Cause

All failing tests hit the same error from CDB/SOS:

Failed to load data access module, 0x80004002
Verify that 1) you have a recent build of the debugger (10.0.18317.1001 or newer)
            2) the file mscordacwks.dll that matches your version of clr.dll is
                in the version directory or on the symbol path

The SOS commands (!PrintException, !ClrStack, !DumpObj, etc.) produce no diagnostic output because the DAC cannot be loaded for the .NET Framework CLR in the debuggee process. The test regex verification then fails because there is no output to match.

This occurs 17 times across the test suite in a single run. The failure is deterministic — identical tests fail on both x64 and x86 with the same error.

Likely Cause

A .NET Framework servicing update on the CI agent machines introduced a new clr.dll version, but the matching mscordacwks.dll is not available on the configured symbol path. See closed issue #4625 for a prior occurrence of missing mscordacwks symbols.

Affected Builds

Build	Date	Failed Legs
1332940	Mar 12	Win x64 Debug, Win x64 Release, Win x86 Release
1332642	Mar 12	Win x64 Debug, Win x64 Release, Win x86 Release
1327950	Mar 10	Win x86 Release

Last passing rolling build: 1330114 (Mar 11).

Failing Tests (15 on x64, 14 on x86)

All desktop.cli config:

• SOS.TaskNestedException, SOS.LineNums, SOS.DivZero, SOS.SimpleThrow, SOS.Reflection
• SOS.StackTraceFaultingExceptionFrame, SOS.GCTests, SOS.OtherCommands
• SOS.NestedExceptionTest, SOS.Overflow, SOS.DumpGen, SOS.ConcurrentDictionaries, SOS.AsyncMain

Plus projectk.sdk.prebuilt config:

• SOS.DualRuntimes (9.0.12), SOS.DualRuntimes (8.0.23)

Reproduction

Artifacts with full test logs are available at:

• Logs_Windows_x64_Release_1 → log/Release/SOS.UnitTests_net8.0_x64.log
• Logs_Windows_x86_Release_1 → log/Release/SOS.UnitTests_net8.0_x86.log

Suggested Fix

1. Verify which .NET Framework version is currently on the CI agents
2. Ensure the symbol server path is configured so CDB can download the matching mscordacwks.dll, or bundle the correct DAC version
3. Alternatively, investigate whether the cdb-sos NuGet package (currently 10.0.26100.1) needs updating to match the new agent environment

[steveisok]

After investigating the test infrastructure in SOSRunner.cs, the root cause is clearer:

The CDB symbol path for desktop.cli tests does not include a symbol server.

In SOSRunner.cs (~line 534), the initial sympath for CDB is set to only the debuggee binary directory:

arguments.AppendFormat(@" -y ""{0}""", debuggeeConfig.BinaryDirPath);

The Microsoft symbol server is not added to the CDB-level sympath for desktop tests. .sympath+ for the runtime symbols path is only configured for triage dumps (line 554–558), and !SetSymbolServer -ms (the SOS-level symbol server) is only added for single-file apps (line 1024).

So when .NET Framework gets a servicing update on the CI agents:

1. CDB sees the new clr.dll in the debuggee process
2. .cordll -l tries to find the matching mscordacwks.dll
3. It only searches the debuggee binary directory — no symbol server
4. → Failed to load data access module, 0x80004002

Possible fix: Add .sympath+ srv*https://msdl.microsoft.com/download/symbols to the initial CDB commands for desktop tests, so CDB can download the matching DAC from the Microsoft symbol server. This would make the tests resilient to .NET Framework servicing updates on CI machines.

Alternatively, the .cordll -lp path could be pointed at the local .NET Framework installation directory where mscordacwks.dll lives alongside clr.dll.

[steveisok]

Updated Root Cause: CDB SecureLoadDotNetExtensions

After the initial PR still showed failures, I dug deeper into the CI logs and found the actual root cause.

What's Really Happening

CDB 10.0.26100.1 (the cdb-sos NuGet package) introduced SecureLoadDotNetExtensions, a security feature that verifies digital signatures of DAC DLLs before loading them. The test infrastructure at SOSRunner.cs:568 was conditionally enabling this:

bool shouldVerifyDacSignature = !config.IsPrivateBuildTesting()
                                && !config.IsNightlyBuild()
                                && !"-none".Equals(config.SetHostRuntime(), ...);
initialCommands.Add($"dx @Debugger.Settings.EngineInitialization.SecureLoadDotNetExtensions={...}");

For rolling CI builds (non-nightly, non-private), this evaluates to true. CDB then rejects DAC DLLs because their signatures cannot be verified:

Failed to verify signature of file: C:\Windows\Microsoft.Net\Framework64\v4.0.30319\mscordacwks.dll
Error code - 0x000021BE
Debugger.Settings.EngineInitialization.SecureLoadDotNetExtensions is enabled, set to false to disable.

Even after adding the Microsoft symbol server to the sympath (so CDB could download mscordacwks.dll), the downloaded copy also fails signature verification:

Failed to verify signature of file: ...\sym\mscordacwks_AMD64_AMD64_4.8.9037.00.dll\...\mscordacwks_AMD64_AMD64_4.8.9037.00.dll

Affected Tests

Category	Count	Why
desktop.cli	13	.NET Framework DAC rejected
projectk.sdk.prebuilt (DualRuntimes)	2	Desktop CLR in dual-runtime dump rejected
projectk.cli.singlefile (8.0.23)	1	.NET 8 servicing DAC rejected

Nightly runtime tests pass because shouldVerifyDacSignature = false when IsNightlyBuild() = true.

Fix

PR #5758 (updated) disables SecureLoadDotNetExtensions unconditionally for CDB and DacSignatureVerification for dotnet-dump. Can be re-enabled once the DAC signing compatibility with CDB 10.0.26100.1 is resolved.

[noahfalk]

As long as SOS is debugging a live debuggee or a dump that was generated on the same machine then mscordacwks.dll is already on the machine in same directory where clr.dll is. I'd recommend using that one rather than adding a symbol server dependency. Network dependencies are slower and less reliable.

[steveisok]

As long as SOS is debugging a live debuggee or a dump that was generated on the same machine then mscordacwks.dll is already on the machine in same directory where clr.dll is. I'd recommend using that one rather than adding a symbol server dependency. Network dependencies are slower and less reliable.

I agree. I think of this as a stop-gap until we fix the actual problem.