CI flakiness: mono interpreter build getting killed

[lewing]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=998900&view=logs&j=3e93a48f-6568-53f2-76b3-1285c183c39d
Build error leg or test failing: Build / browser-wasm linux Release AllSubsets_Mono_RuntimeTests monointerpreter / Build Tests
Pull request: #114058

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "ErrorPattern": "RuntimeVariant=monointerpreter.*exited with code 137",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=998900
Error message validated: [RuntimeVariant=monointerpreter*exited with code 137.]
Result validation: ❌ Known issue did not match with the provided build.
Validation performed at: 4/1/2025 9:14:37 PM UTC

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=998900
Error message validated: [RuntimeVariant=monointerpreter.*exited with code 137]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 4/1/2025 9:15:42 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1044192	dotnet/runtime	Build Tests	Log	#115544
1043795	dotnet/runtime	Build Tests	Log	#115576
1043787	dotnet/runtime	Build Tests	Log
1043474	dotnet/runtime	Build Tests	Log	#115039
1043408	dotnet/runtime	Build Tests	Log	#115531
1043376	dotnet/runtime	Build Tests	Log	#115576
1043385	dotnet/runtime	Build Tests	Log
1043365	dotnet/runtime	Build Tests	Log
1043357	dotnet/runtime	Build Tests	Log	#115545
1043242	dotnet/runtime	Build Tests	Log	#115408
1043079	dotnet/runtime	Build Tests	Log	#113956
1042975	dotnet/runtime	Build Tests	Log	#115574
1042922	dotnet/runtime	Build Tests	Log	#115591
1042899	dotnet/runtime	Build Tests	Log
1042786	dotnet/runtime	Build Tests	Log	#115574
1042728	dotnet/runtime	Build Tests	Log	#115549
1042673	dotnet/runtime	Build Tests	Log	#114438
1042459	dotnet/runtime	Build Tests	Log	#115573
1042605	dotnet/runtime	Build Tests	Log	#115058
1042420	dotnet/runtime	Build Tests	Log	#115563
1042578	dotnet/runtime	Build Tests	Log	#115574
1042552	dotnet/runtime	Build Tests	Log	#115610
1042330	dotnet/runtime	Build Tests	Log	#115408
1042465	dotnet/runtime	Build Tests	Log	#115605
1042414	dotnet/runtime	Build Tests	Log	#115427
1042349	dotnet/runtime	Build Tests	Log	#115573
1041448	dotnet/runtime	Build Tests	Log	#115553
1042324	dotnet/runtime	Build Tests	Log	#113956
1041806	dotnet/runtime	Build Tests	Log	#115494
1042282	dotnet/runtime	Build Tests	Log	#115251
1042106	dotnet/runtime	Build Tests	Log	#115584
1042019	dotnet/runtime	Build Tests	Log	#115375
1041906	dotnet/runtime	Build Tests	Log	#115544
1041632	dotnet/runtime	Build Tests	Log	#115563
1041747	dotnet/runtime	Build Tests	Log	#115584
1041706	dotnet/runtime	Build Tests	Log	#115580
1041705	dotnet/runtime	Build Tests	Log
1041628	dotnet/runtime	Build Tests	Log	#115573
1041490	dotnet/runtime	Build Tests	Log	#115408
1041509	dotnet/runtime	Build Tests	Log	#115568
1041497	dotnet/runtime	Build Tests	Log	#115566
1041461	dotnet/runtime	Build Tests	Log	#115335
1041453	dotnet/runtime	Build Tests	Log	#115563
1041435	dotnet/runtime	Build Tests	Log	#114148
1041427	dotnet/runtime	Build Tests	Log	#115113
1041398	dotnet/runtime	Build Tests	Log	#115375
1041288	dotnet/runtime	Build Tests	Log	#115556
1041350	dotnet/runtime	Build Tests	Log	#114148
1041343	dotnet/runtime	Build Tests	Log	#115408
1041306	dotnet/runtime	Build Tests	Log	#115553
1041309	dotnet/runtime	Build Tests	Log	#115513
1041259	dotnet/runtime	Build Tests	Log	#114438
1041224	dotnet/runtime	Build Tests	Log	#115408
1041207	dotnet/runtime	Build Tests	Log
1041158	dotnet/runtime	Build Tests	Log	#115549
1040895	dotnet/runtime	Build Tests	Log	#115538
1040841	dotnet/runtime	Build Tests	Log	#114438
1040823	dotnet/runtime	Build Tests	Log	#115375
1040764	dotnet/runtime	Build Tests	Log	#114881
1040707	dotnet/runtime	Build Tests	Log
1040533	dotnet/runtime	Build Tests	Log	#115375
1040530	dotnet/runtime	Build Tests	Log	#114881
1040520	dotnet/runtime	Build Tests	Log	#115516
1040497	dotnet/runtime	Build Tests	Log
1040339	dotnet/runtime	Build Tests	Log
1040229	dotnet/runtime	Build Tests	Log	#115408
1040208	dotnet/runtime	Build Tests	Log	#115465
1040024	dotnet/runtime	Build Tests	Log	#114881
1040013	dotnet/runtime	Build Tests	Log	#115497
1039991	dotnet/runtime	Build Tests	Log	#115494
1039973	dotnet/runtime	Build Tests	Log
1039922	dotnet/runtime	Build Tests	Log	#115436
1039916	dotnet/runtime	Build Tests	Log	#115437
1039841	dotnet/runtime	Build Tests	Log
1039786	dotnet/runtime	Build Tests	Log	#115443
1039768	dotnet/runtime	Build Tests	Log	#114901
1039606	dotnet/runtime	Build Tests	Log
1039581	dotnet/runtime	Build Tests	Log	#115335
1039493	dotnet/runtime	Build Tests	Log	#115408
1039394	dotnet/runtime	Build Tests	Log	#115428
1039386	dotnet/runtime	Build Tests	Log	#114901
1039324	dotnet/runtime	Build Tests	Log	#115335
1039302	dotnet/runtime	Build Tests	Log
1038979	dotnet/runtime	Build Tests	Log
1038946	dotnet/runtime	Build Tests	Log	#115408
1038942	dotnet/runtime	Build Tests	Log	#115443
1038865	dotnet/runtime	Build Tests	Log
1038844	dotnet/runtime	Build Tests	Log	#115446
1038838	dotnet/runtime	Build Tests	Log	#114597
1038813	dotnet/runtime	Build Tests	Log	#115441
1038758	dotnet/runtime	Build Tests	Log	#115436
1038725	dotnet/runtime	Build Tests	Log
1038719	dotnet/runtime	Build Tests	Log	#115408
1038652	dotnet/runtime	Build Tests	Log	#115444
1038645	dotnet/runtime	Build Tests	Log	#114881
1038616	dotnet/runtime	Build Tests	Log	#115388
1038527	dotnet/runtime	Build Tests	Log
1038464	dotnet/runtime	Build Tests	Log	#115444
1038448	dotnet/runtime	Build Tests	Log	#115443
1038374	dotnet/runtime	Build Tests	Log	#115441
Displaying 100 of 273 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	82	273

[dotnet-policy-service]

Tagging subscribers to this area: @akoeplinger, @matouskozak
See info in area-owners.md if you want to be subscribed.

[matouskozak]

fyi: @BrzVlad

[pavelsavara]

We use mono interpreter to run msbuild, right ?

This is building 3 groups of many small tests

That combined with WASM toolchain memory demands makes msbuild run out of memory (docker kills it).
This is OOM of the msbuild, not OOM of the individual test projects.
I think this is OOM kill, not timeout because the duration before the kill fluctuates.

As far as I can tell, it never passes for WASM and it's killed in the group 2 already.

Log

2025-05-15T23:09:37.6650337Z   Number of test projects in group 2: 2366
2025-05-15T23:09:37.7341670Z   Building managed test group 2: "/__w/1/s/dotnet.sh" msbuild /__w/1/s/src/tests/build.proj /t:Build "/p:TargetArchitecture=wasm" "/p:Configuration=Release" "/p:LibrariesConfiguration=Release" "/p:TasksConfiguration=Release" "/p:TargetOS=browser" "/p:ToolsOS=" "/p:PackageOS=" "/p:RuntimeFlavor=mono" "/p:RuntimeVariant=monointerpreter" "/p:CLRTestBuildAllTargets=" "/p:UseCodeFlowEnforcement=" "/p:__TestGroupToBuild=2" "/p:__SkipRestorePackages=1" /nodeReuse:false /m:1 /bl:/__w/1/s/artifacts//log/Release/InnerManagedTestBuild.2.binlog /p:ContinuousIntegrationBuild=true "/p:DevTeamProvisioning=-"
2025-05-15T23:09:37.9271786Z   MSBuild version 17.14.0-preview-25177-05+903614e16 for .NET
2025-05-15T23:09:37.9852691Z   
2025-05-15T23:09:40.5433817Z     Number of test projects in group 2: 2366
2025-05-15T23:09:37.7341670Z   Building managed test group 2: "/__w/1/s/dotnet.sh" msbuild /__w/1/s/src/tests/build.proj /t:Build "/p:TargetArchitecture=wasm" "/p:Configuration=Release" "/p:LibrariesConfiguration=Release" "/p:TasksConfiguration=Release" "/p:TargetOS=browser" "/p:ToolsOS=" "/p:PackageOS=" "/p:RuntimeFlavor=mono" "/p:RuntimeVariant=monointerpreter" "/p:CLRTestBuildAllTargets=" "/p:UseCodeFlowEnforcement=" "/p:__TestGroupToBuild=2" "/p:__SkipRestorePackages=1" /nodeReuse:false /m:1 /bl:/__w/1/s/artifacts//log/Release/InnerManagedTestBuild.2.binlog /p:ContinuousIntegrationBuild=true "/p:DevTeamProvisioning=-"
2025-05-15T23:09:37.9271786Z   MSBuild version 17.14.0-preview-25177-05+903614e16 for .NET
2025-05-15T23:09:37.9852691Z   
2025-05-15T23:09:40.5433817Z     Number of test projects in group 2: 2366
2025-05-15T23:09:41.8292504Z     XUnitWrapperGenerator -> /__w/1/s/artifacts/tests/coreclr/browser.wasm.Release/Common/XUnitWrapperGenerator/XUnitWrapperGenerator/XUnitWrapperGenerator.dll
2025-05-15T23:09:42.6596953Z     KeepAliveBoxOpt -> /__w/1/s/artifacts/tests/coreclr/browser.wasm.Release/JIT/Methodical/Boxing/boxunbox/KeepAliveBoxOpt/KeepAliveBoxOpt.dll
2025-05-15T23:09:42.9728086Z     sinlib_cs -> /__w/1/s/artifacts/tests/coreclr/browser.wasm.Release/JIT/Methodical/Boxing/xlang/sinlib_cs/sinlib_cs.dll
2025-05-15T23:09:43.2633097Z     testlib_misc -> /__w/1/s/artifacts/tests/coreclr/browser.wasm.Release/JIT/Methodical/cctor/misc/testlib_misc/testlib_misc.dll
2025-05-15T23:09:43.6004947Z     threads1_cs_do -> /__w/1/s/artifacts/tests/coreclr/browser.wasm.Release/JIT/Methodical/cctor/misc/threads1_cs_do/threads1_cs_do.dll
2025-05-15T23:09:43.8861163Z     testlib_xassem -> /__w/1/s/artifacts/tests/coreclr/browser.wasm.Release/JIT/Methodical/cctor/xassem/testlib_xassem/testlib_xassem.dll
...
2025-05-15T23:23:01.9327210Z     b53977 -> /__w/1/s/artifacts/tests/coreclr/browser.wasm.Release/JIT/Regression/CLR-x86-JIT/V1-M12-Beta2/b53977/b53977/b53977.dll
2025-05-15T23:23:08.5186861Z   /__w/1/s/dotnet.sh: line 27: 50331 Killed                  ${dotnetPath} "$@"
2025-05-15T23:23:08.5814254Z ##[error]src/tests/build.proj(501,5): error MSB3073: The command ""/__w/1/s/dotnet.sh" msbuild /__w/1/s/src/tests/build.proj /t:Build "/p:TargetArchitecture=wasm" "/p:Configuration=Release" "/p:LibrariesConfiguration=Release" "/p:TasksConfiguration=Release" "/p:TargetOS=browser" "/p:ToolsOS=" "/p:PackageOS=" "/p:RuntimeFlavor=mono" "/p:RuntimeVariant=monointerpreter" "/p:CLRTestBuildAllTargets=" "/p:UseCodeFlowEnforcement=" "/p:__TestGroupToBuild=2" "/p:__SkipRestorePackages=1" /nodeReuse:false /m:1 /bl:/__w/1/s/artifacts//log/Release/InnerManagedTestBuild.2.binlog /p:ContinuousIntegrationBuild=true "/p:DevTeamProvisioning=-"" exited with code 137.

[pavelsavara]

I disabled part of the test on browser, because 100% fail rate was not helping anybody. This is ActiveIssue now.