[outerloop] System.Runtime.Tests crash with exit code 137 in linux debug coreclr

[carlossanlop]

Found in a PR that fixed an unrelated failure affecting all outerloop runs, hiding other failures. #97135

Error Blob

{
  "ErrorMessage": "",
  "BuildRetry": false,
  "ErrorPattern": "Killed.+System\\.Runtime\\.Tests\\.runtimeconfig\\.json.+OuterLoop",
  "ExcludeConsoleLog": false
}

Reproduction Steps

• Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=580195
• Queues:
  • linux_musl-x64 Debug CoreCLR_Debug
  • linux-x64 Debug CoreCLR_Debug
• Job results:
  • https://dev.azure.com/dnceng-public/public/_build/results?buildId=580195&view=logs&j=592eed89-78f5-5202-9da5-ad17df3bef4a&t=4feb010c-08b1-5201-8e2c-97e3caf24223
  • https://dev.azure.com/dnceng-public/public/_build/results?buildId=580195&view=logs&j=df56ac33-e9fe-5eb2-5cdb-3303d7aad000&t=f62247f8-d051-5cb6-d5f0-06f532f9a005
• Log files:
  • https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-97135-merge-6fc0e798b0d84fd99e/System.Runtime.Tests/1/console.c8ffe024.log?helixlogtype=result
  • https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-97135-merge-ca234b0b7f3e48249a/System.Runtime.Tests/1/console.40eef85c.log?helixlogtype=result
• Output:

========================== End custom configuration settings ===============================
----- start Mon Feb 26 23:56:03 UTC 2024 =============== To repro directly: =====================================================
pushd .
/root/helix/work/correlation/dotnet exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -nocolor -trait category=OuterLoop -notrait category=AdditionalTimezoneChecks -notrait category=IgnoreForCI -notrait category=failing 
popd
===========================================================================================================
/root/helix/work/workitem/e /root/helix/work/workitem/e
  Discovering: System.Runtime.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Runtime.Tests (found 29 of 9308 test cases)
  Starting:    System.Runtime.Tests (parallel test collections = on [2 threads], stop on fail = off)
./RunTests.sh: line 180:    21 Killed                  "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -nocolor -trait category=OuterLoop -notrait category=AdditionalTimezoneChecks -notrait category=IgnoreForCI -notrait category=failing $RSP_FILE
/root/helix/work/workitem/e
----- end Mon Feb 26 23:56:28 UTC 2024 ----- exit code 137 ----------------------------------------------------------
exit code 137 means SIGKILL Killed either due to out of memory/resources (see /var/log/messages) or by explicit kill.
ulimit -c value: unlimited
cat /proc/sys/kernel/core_pattern: /home/helixbot/dotnetbuild/dumps/core.%u.%p
cat /proc/sys/kernel/core_uses_pid: 1
cat: /proc/sys/kernel/coredump_filter: No such file or directory
cat /proc/sys/kernel/coredump_filter:
Looking around for any Linux dumps...
Looking for files matching core.* ...
----- start =============== XUnitLogChecker Output =====================================================
dmesg: klogctl: Operation not permitted
No dumps found in /home/helixbot/dotnetbuild/dumps.
----- end =============== XUnitLogChecker Output - exit code 0 ===========================
+ export '_commandExitCode=137'
+ python /root/helix/work/correlation/reporter/run.py https://dev.azure.com/dnceng-public/ public 13931354 eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Im9PdmN6NU1fN3AtSGpJS2xGWHo5M3VfVjBabyJ9.eyJuYW1laWQiOiJjNzczZjJjMi01MTIwLTQyMDctYWZlMi1hZmFmMzVhOGJjMGEiLCJzY3AiOiJhcHBfdG9rZW4iLCJhdWkiOiJhMTBlYjk0MS0xNzM2LTRjNzYtOWFiZC1iZDBlZGYxNjg0ODgiLCJzaWQiOiIxZTViN2VlMC1iNmQyLTQzNjgtODZjOS0zOTQ5MGIxM2FlODAiLCJCdWlsZElkIjoiY2JiMTgyNjEtYzQ4Zi00YWJiLTg2NTEtOGNkY2I1NDc0NjQ5OzU4MDE5NSIsImpvYnJlZiI6IjMxNTZhMmQwLTExYWItNGExMC04NTA0LWY3Y2UxM2ZiMTk2OTo1OTJlZWQ4OS03OGY1LTUyMDItOWRhNS1hZDE3ZGYzYmVmNGEiLCJwcGlkIjoidnN0ZnM6Ly8vQnVpbGQvQnVpbGQvNTgwMTk1Iiwib3JjaGlkIjoiMzE1NmEyZDAtMTFhYi00YTEwLTg1MDQtZjdjZTEzZmIxOTY5LmJ1aWxkLmJ1aWxkX2xpbnV4X211c2xfeDY0X2RlYnVnX2NvcmVjbHJfZGVidWcuX19kZWZhdWx0IiwicmVwb0lkcyI6IiIsImlzcyI6ImFwcC52c3Rva2VuLnZpc3VhbHN0dWRpby5jb20iLCJhdWQiOiJhcHAudnN0b2tlbi52aXN1YWxzdHVkaW8uY29tfHZzbzo2ZmNjOTJlNS03M2E3LTRmODgtOGQxMy1kOTA0NWI0NWZiMjciLCJuYmYiOjE3MDg5ODg5MzgsImV4cCI6MTcwOTAwMDkzOH0.rjlcWYrlHLfuHwUtUIiDkhhUDwy8cnQZ3rCBJjmd8FuCN0BDFzjz0jEMQEbQ9WZ7LHqdWN40vrD1MHH2p9a2tC9JYY7ca4b2w8Yy05wlbG-_cjF_ksT3jfhgz4EOLBiSmE7-bW_m1b3gtKHo-rdFDyMt_vciLiIB_C3543QhcPGJJdph5ok1SjBVKbl1q9wzw4Tut1lLyo8-ToLRotVPvv2Fahz3TSp--EDbdhY2SYtNrPyGVW7t9F8o3TZdG5iyrtyTnCAlr2Qdu-cOUJhyXM_4vhztXvLdgDMuDgGINPpA7sDXoqo_7LB9RP1iNc35uIEM-430j7jeG9MDrZzEZQ
2024-02-26T23:56:35.053Z	INFO   	run.py	run(48)	main	Beginning reading of test results.
2024-02-26T23:56:35.054Z	INFO   	run.py	__init__(42)	read_results	Searching '/root/helix/work/workitem/e' for test results files
2024-02-26T23:56:35.054Z	INFO   	run.py	__init__(42)	read_results	Searching '/root/helix/work/workitem/uploads' for test results files
2024-02-26T23:56:35.062Z	WARNING	run.py	__init__(55)	read_results	No results file found in any of the following formats: xunit, junit, trx
2024-02-26T23:56:35.063Z	INFO   	run.py	packing_test_reporter(30)	report_results	Packing 0 test reports to '/root/helix/work/workitem/e/__test_report.json'
2024-02-26T23:56:35.067Z	INFO   	run.py	packing_test_reporter(33)	report_results	Packed 1541 bytes
+ python /root/helix/work/correlation/gen-debug-dump-docs.py -buildid 580195 -workitem System.Runtime.Tests -jobid 6fc0e798-b0d8-4fd9-9e43-a36cfb9441c4 -outdir /root/helix/work/workitem/uploads -templatedir /root/helix/work/correlation -dumpdir /home/helixbot/dotnetbuild/dumps -productver 9.0.0
gen-debug-dump-docs.py: Did not find dumps, skipping dump docs generation.
+ exit 137
+ export '_commandExitCode=137'
+ chmod -R 777 /home/helixbot/dotnetbuild/dumps
+ exit 137

[END EXECUTION]
Exit Code:137

Report

Build	Definition	Test	Pull Request
1084947	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1084939	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1083878	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1083881	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1083222	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#115502
1082773	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1082762	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1081821	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1081824	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1081486	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#115502
1081257	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1081259	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1081136	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#115502
1081093	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#115502
1080967	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1080959	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1080004	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1080003	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1078741	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1078737	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1077770	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1077768	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1076637	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1076642	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1075440	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1075437	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1074995	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1074986	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1074699	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1074706	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1073663	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1073655	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1072448	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1072453	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1071322	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1071320	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1070065	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1070066	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1068943	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1068936	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1068511	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1068509	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1068276	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1068274	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1067461	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#116631
1067326	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1067318	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1066069	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1066060	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1064793	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1064800	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1063509	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1063507	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1062370	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1062372	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1061926	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1061920	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1061631	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1061632	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1061257	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#115502
1060886	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1060891	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1059893	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1059890	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1058725	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
1058720	dotnet/runtime	System.Runtime.Tests.WorkItemExecution

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
2	18	66

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=580195
Error message validated: [Killed.+System\.Runtime\.Tests\.runtimeconfig\.json.+OuterLoop]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/27/2024 11:09:08 PM UTC

Tagging subscribers to this area: @dotnet/area-system-runtime
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Error Blob

{
  "ErrorMessage": "",
  "BuildRetry": false,
  "ErrorPattern": "Killed.+System\.Runtime\.Tests\.runtimeconfig\.json.+OuterLoop",
  "ExcludeConsoleLog": false
}

Reproduction Steps

• Queues:
  • linux_musl-x64 Debug CoreCLR_Debug
  • linux-x64 Debug CoreCLR_Debug
• Job results:
  • https://dev.azure.com/dnceng-public/public/_build/results?buildId=580195&view=logs&j=592eed89-78f5-5202-9da5-ad17df3bef4a&t=4feb010c-08b1-5201-8e2c-97e3caf24223
  • https://dev.azure.com/dnceng-public/public/_build/results?buildId=580195&view=logs&j=df56ac33-e9fe-5eb2-5cdb-3303d7aad000&t=f62247f8-d051-5cb6-d5f0-06f532f9a005
• Log files:
  • https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-97135-merge-6fc0e798b0d84fd99e/System.Runtime.Tests/1/console.c8ffe024.log?helixlogtype=result
  • https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-97135-merge-ca234b0b7f3e48249a/System.Runtime.Tests/1/console.40eef85c.log?helixlogtype=result
• Output:

========================== End custom configuration settings ===============================
----- start Mon Feb 26 23:56:03 UTC 2024 =============== To repro directly: =====================================================
pushd .
/root/helix/work/correlation/dotnet exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -nocolor -trait category=OuterLoop -notrait category=AdditionalTimezoneChecks -notrait category=IgnoreForCI -notrait category=failing 
popd
===========================================================================================================
/root/helix/work/workitem/e /root/helix/work/workitem/e
  Discovering: System.Runtime.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Runtime.Tests (found 29 of 9308 test cases)
  Starting:    System.Runtime.Tests (parallel test collections = on [2 threads], stop on fail = off)
./RunTests.sh: line 180:    21 Killed                  "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -nocolor -trait category=OuterLoop -notrait category=AdditionalTimezoneChecks -notrait category=IgnoreForCI -notrait category=failing $RSP_FILE
/root/helix/work/workitem/e
----- end Mon Feb 26 23:56:28 UTC 2024 ----- exit code 137 ----------------------------------------------------------
exit code 137 means SIGKILL Killed either due to out of memory/resources (see /var/log/messages) or by explicit kill.
ulimit -c value: unlimited
cat /proc/sys/kernel/core_pattern: /home/helixbot/dotnetbuild/dumps/core.%u.%p
cat /proc/sys/kernel/core_uses_pid: 1
cat: /proc/sys/kernel/coredump_filter: No such file or directory
cat /proc/sys/kernel/coredump_filter:
Looking around for any Linux dumps...
Looking for files matching core.* ...
----- start =============== XUnitLogChecker Output =====================================================
dmesg: klogctl: Operation not permitted
No dumps found in /home/helixbot/dotnetbuild/dumps.
----- end =============== XUnitLogChecker Output - exit code 0 ===========================
+ export '_commandExitCode=137'
+ python /root/helix/work/correlation/reporter/run.py https://dev.azure.com/dnceng-public/ public 13931354 eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Im9PdmN6NU1fN3AtSGpJS2xGWHo5M3VfVjBabyJ9.eyJuYW1laWQiOiJjNzczZjJjMi01MTIwLTQyMDctYWZlMi1hZmFmMzVhOGJjMGEiLCJzY3AiOiJhcHBfdG9rZW4iLCJhdWkiOiJhMTBlYjk0MS0xNzM2LTRjNzYtOWFiZC1iZDBlZGYxNjg0ODgiLCJzaWQiOiIxZTViN2VlMC1iNmQyLTQzNjgtODZjOS0zOTQ5MGIxM2FlODAiLCJCdWlsZElkIjoiY2JiMTgyNjEtYzQ4Zi00YWJiLTg2NTEtOGNkY2I1NDc0NjQ5OzU4MDE5NSIsImpvYnJlZiI6IjMxNTZhMmQwLTExYWItNGExMC04NTA0LWY3Y2UxM2ZiMTk2OTo1OTJlZWQ4OS03OGY1LTUyMDItOWRhNS1hZDE3ZGYzYmVmNGEiLCJwcGlkIjoidnN0ZnM6Ly8vQnVpbGQvQnVpbGQvNTgwMTk1Iiwib3JjaGlkIjoiMzE1NmEyZDAtMTFhYi00YTEwLTg1MDQtZjdjZTEzZmIxOTY5LmJ1aWxkLmJ1aWxkX2xpbnV4X211c2xfeDY0X2RlYnVnX2NvcmVjbHJfZGVidWcuX19kZWZhdWx0IiwicmVwb0lkcyI6IiIsImlzcyI6ImFwcC52c3Rva2VuLnZpc3VhbHN0dWRpby5jb20iLCJhdWQiOiJhcHAudnN0b2tlbi52aXN1YWxzdHVkaW8uY29tfHZzbzo2ZmNjOTJlNS03M2E3LTRmODgtOGQxMy1kOTA0NWI0NWZiMjciLCJuYmYiOjE3MDg5ODg5MzgsImV4cCI6MTcwOTAwMDkzOH0.rjlcWYrlHLfuHwUtUIiDkhhUDwy8cnQZ3rCBJjmd8FuCN0BDFzjz0jEMQEbQ9WZ7LHqdWN40vrD1MHH2p9a2tC9JYY7ca4b2w8Yy05wlbG-_cjF_ksT3jfhgz4EOLBiSmE7-bW_m1b3gtKHo-rdFDyMt_vciLiIB_C3543QhcPGJJdph5ok1SjBVKbl1q9wzw4Tut1lLyo8-ToLRotVPvv2Fahz3TSp--EDbdhY2SYtNrPyGVW7t9F8o3TZdG5iyrtyTnCAlr2Qdu-cOUJhyXM_4vhztXvLdgDMuDgGINPpA7sDXoqo_7LB9RP1iNc35uIEM-430j7jeG9MDrZzEZQ
2024-02-26T23:56:35.053Z	INFO   	run.py	run(48)	main	Beginning reading of test results.
2024-02-26T23:56:35.054Z	INFO   	run.py	__init__(42)	read_results	Searching '/root/helix/work/workitem/e' for test results files
2024-02-26T23:56:35.054Z	INFO   	run.py	__init__(42)	read_results	Searching '/root/helix/work/workitem/uploads' for test results files
2024-02-26T23:56:35.062Z	WARNING	run.py	__init__(55)	read_results	No results file found in any of the following formats: xunit, junit, trx
2024-02-26T23:56:35.063Z	INFO   	run.py	packing_test_reporter(30)	report_results	Packing 0 test reports to '/root/helix/work/workitem/e/__test_report.json'
2024-02-26T23:56:35.067Z	INFO   	run.py	packing_test_reporter(33)	report_results	Packed 1541 bytes
+ python /root/helix/work/correlation/gen-debug-dump-docs.py -buildid 580195 -workitem System.Runtime.Tests -jobid 6fc0e798-b0d8-4fd9-9e43-a36cfb9441c4 -outdir /root/helix/work/workitem/uploads -templatedir /root/helix/work/correlation -dumpdir /home/helixbot/dotnetbuild/dumps -productver 9.0.0
gen-debug-dump-docs.py: Did not find dumps, skipping dump docs generation.
+ exit 137
+ export '_commandExitCode=137'
+ chmod -R 777 /home/helixbot/dotnetbuild/dumps
+ exit 137

[END EXECUTION]
Exit Code:137

Author:	carlossanlop
Assignees:	-
Labels:	area-System.Runtime, os-linux, arch-x64, blocking-clean-ci, runtime-coreclr, os-linux-musl, Known Build Error
Milestone:	-

[danmoseley]

dmesg: read kernel buffer failed: Operation not permitted

Again.. so we can't see whether it was OOM. But presumably it was

[ilonatommy]

Hit on net9.0-linux-Debug-x64-coreclr_release queue, LibraryImportGenerator.Tests but not caught by the query:
https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-99685-merge-4e96f500c0fc4768b9/LibraryImportGenerator.Tests/1/console.9506f96c.log?helixlogtype=result

./RunTests.sh: line 180:    24 Killed                  "$RUNTIME_PATH/dotnet" exec --runtimeconfig LibraryImportGenerator.Tests.runtimeconfig.json --depsfile LibraryImportGenerator.Tests.deps.json xunit.console.dll LibraryImportGenerator.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
----- end Wed Apr 3 08:14:35 UTC 2024 ----- exit code 137 ----------------------------------------------------------
exit code 137 means SIGKILL Killed either due to out of memory/resources (see /var/log/messages) or by explicit kill.
ulimit -c value: unlimited
cat /proc/sys/kernel/core_pattern: /home/helixbot/dotnetbuild/dumps/core.%u.%p
cat /proc/sys/kernel/core_uses_pid: 1
cat: /proc/sys/kernel/coredump_filter: No such file or directory
cat /proc/sys/kernel/coredump_filter:
Looking around for any Linux dumps...
Looking for files matching core.* ...
dmesg: read kernel buffer failed: Operation not permitted

[pavelsavara]

again https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-98492-merge-31c70303cf32472f92/LibraryImportGenerator.Tests/1/console.c806de40.log?helixlogtype=result

[pavelsavara]

should we edit the filter here to match exit code 137 ? It didn't found and reported LibraryImportGenerator.Tests in build