Skip to content

[outerloop] System.Runtime.Tests crash with exit code 137 in linux debug coreclr #98964

Open
@carlossanlop

Description

@carlossanlop

Found in a PR that fixed an unrelated failure affecting all outerloop runs, hiding other failures. #97135

Error Blob

{
  "ErrorMessage": "",
  "BuildRetry": false,
  "ErrorPattern": "Killed.+System\\.Runtime\\.Tests\\.runtimeconfig\\.json.+OuterLoop",
  "ExcludeConsoleLog": false
}

Reproduction Steps

========================== End custom configuration settings ===============================
----- start Mon Feb 26 23:56:03 UTC 2024 =============== To repro directly: =====================================================
pushd .
/root/helix/work/correlation/dotnet exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -nocolor -trait category=OuterLoop -notrait category=AdditionalTimezoneChecks -notrait category=IgnoreForCI -notrait category=failing 
popd
===========================================================================================================
/root/helix/work/workitem/e /root/helix/work/workitem/e
  Discovering: System.Runtime.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Runtime.Tests (found 29 of 9308 test cases)
  Starting:    System.Runtime.Tests (parallel test collections = on [2 threads], stop on fail = off)
./RunTests.sh: line 180:    21 Killed                  "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -nocolor -trait category=OuterLoop -notrait category=AdditionalTimezoneChecks -notrait category=IgnoreForCI -notrait category=failing $RSP_FILE
/root/helix/work/workitem/e
----- end Mon Feb 26 23:56:28 UTC 2024 ----- exit code 137 ----------------------------------------------------------
exit code 137 means SIGKILL Killed either due to out of memory/resources (see /var/log/messages) or by explicit kill.
ulimit -c value: unlimited
cat /proc/sys/kernel/core_pattern: /home/helixbot/dotnetbuild/dumps/core.%u.%p
cat /proc/sys/kernel/core_uses_pid: 1
cat: /proc/sys/kernel/coredump_filter: No such file or directory
cat /proc/sys/kernel/coredump_filter:
Looking around for any Linux dumps...
Looking for files matching core.* ...
----- start =============== XUnitLogChecker Output =====================================================
dmesg: klogctl: Operation not permitted
No dumps found in /home/helixbot/dotnetbuild/dumps.
----- end =============== XUnitLogChecker Output - exit code 0 ===========================
+ export '_commandExitCode=137'
+ python /root/helix/work/correlation/reporter/run.py https://dev.azure.com/dnceng-public/ public 13931354 eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Im9PdmN6NU1fN3AtSGpJS2xGWHo5M3VfVjBabyJ9.eyJuYW1laWQiOiJjNzczZjJjMi01MTIwLTQyMDctYWZlMi1hZmFmMzVhOGJjMGEiLCJzY3AiOiJhcHBfdG9rZW4iLCJhdWkiOiJhMTBlYjk0MS0xNzM2LTRjNzYtOWFiZC1iZDBlZGYxNjg0ODgiLCJzaWQiOiIxZTViN2VlMC1iNmQyLTQzNjgtODZjOS0zOTQ5MGIxM2FlODAiLCJCdWlsZElkIjoiY2JiMTgyNjEtYzQ4Zi00YWJiLTg2NTEtOGNkY2I1NDc0NjQ5OzU4MDE5NSIsImpvYnJlZiI6IjMxNTZhMmQwLTExYWItNGExMC04NTA0LWY3Y2UxM2ZiMTk2OTo1OTJlZWQ4OS03OGY1LTUyMDItOWRhNS1hZDE3ZGYzYmVmNGEiLCJwcGlkIjoidnN0ZnM6Ly8vQnVpbGQvQnVpbGQvNTgwMTk1Iiwib3JjaGlkIjoiMzE1NmEyZDAtMTFhYi00YTEwLTg1MDQtZjdjZTEzZmIxOTY5LmJ1aWxkLmJ1aWxkX2xpbnV4X211c2xfeDY0X2RlYnVnX2NvcmVjbHJfZGVidWcuX19kZWZhdWx0IiwicmVwb0lkcyI6IiIsImlzcyI6ImFwcC52c3Rva2VuLnZpc3VhbHN0dWRpby5jb20iLCJhdWQiOiJhcHAudnN0b2tlbi52aXN1YWxzdHVkaW8uY29tfHZzbzo2ZmNjOTJlNS03M2E3LTRmODgtOGQxMy1kOTA0NWI0NWZiMjciLCJuYmYiOjE3MDg5ODg5MzgsImV4cCI6MTcwOTAwMDkzOH0.rjlcWYrlHLfuHwUtUIiDkhhUDwy8cnQZ3rCBJjmd8FuCN0BDFzjz0jEMQEbQ9WZ7LHqdWN40vrD1MHH2p9a2tC9JYY7ca4b2w8Yy05wlbG-_cjF_ksT3jfhgz4EOLBiSmE7-bW_m1b3gtKHo-rdFDyMt_vciLiIB_C3543QhcPGJJdph5ok1SjBVKbl1q9wzw4Tut1lLyo8-ToLRotVPvv2Fahz3TSp--EDbdhY2SYtNrPyGVW7t9F8o3TZdG5iyrtyTnCAlr2Qdu-cOUJhyXM_4vhztXvLdgDMuDgGINPpA7sDXoqo_7LB9RP1iNc35uIEM-430j7jeG9MDrZzEZQ
2024-02-26T23:56:35.053Z	INFO   	run.py	run(48)	main	Beginning reading of test results.
2024-02-26T23:56:35.054Z	INFO   	run.py	__init__(42)	read_results	Searching '/root/helix/work/workitem/e' for test results files
2024-02-26T23:56:35.054Z	INFO   	run.py	__init__(42)	read_results	Searching '/root/helix/work/workitem/uploads' for test results files
2024-02-26T23:56:35.062Z	WARNING	run.py	__init__(55)	read_results	No results file found in any of the following formats: xunit, junit, trx
2024-02-26T23:56:35.063Z	INFO   	run.py	packing_test_reporter(30)	report_results	Packing 0 test reports to '/root/helix/work/workitem/e/__test_report.json'
2024-02-26T23:56:35.067Z	INFO   	run.py	packing_test_reporter(33)	report_results	Packed 1541 bytes
+ python /root/helix/work/correlation/gen-debug-dump-docs.py -buildid 580195 -workitem System.Runtime.Tests -jobid 6fc0e798-b0d8-4fd9-9e43-a36cfb9441c4 -outdir /root/helix/work/workitem/uploads -templatedir /root/helix/work/correlation -dumpdir /home/helixbot/dotnetbuild/dumps -productver 9.0.0
gen-debug-dump-docs.py: Did not find dumps, skipping dump docs generation.
+ exit 137
+ export '_commandExitCode=137'
+ chmod -R 777 /home/helixbot/dotnetbuild/dumps
+ exit 137

[END EXECUTION]
Exit Code:137

Report

Build Definition Test Pull Request
1044562 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1044560 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1044302 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1044297 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1043487 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1043481 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1042446 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1042444 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1041302 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1041299 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1040398 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1040401 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1040128 dotnet/runtime System.Runtime.Tests.WorkItemExecution #115502
1039436 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1039434 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1038884 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1038882 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1038559 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1038561 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1037692 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1037690 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1036855 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1036857 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1035683 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1035682 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1035079 dotnet/runtime System.Runtime.Tests.WorkItemExecution #114523
1034608 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1034607 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1032904 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1032903 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1032214 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1032216 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1031430 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1031431 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1029591 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1029590 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1028245 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1028244 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1027732 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1027733 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1027351 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1027349 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1026485 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1026486 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1025333 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1025332 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1025326 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1025330 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1023988 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1024008 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1023983 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1023986 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1022740 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1022744 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1022759 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1022741 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1021773 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1021772 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1021746 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1021744 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1021293 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1021288 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1021289 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1021281 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1021062 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1021046 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1021043 dotnet/runtime System.Runtime.Tests.WorkItemExecution
1021052 dotnet/runtime System.Runtime.Tests.WorkItemExecution

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
2 15 68

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=580195
Error message validated: [Killed.+System\.Runtime\.Tests\.runtimeconfig\.json.+OuterLoop]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/27/2024 11:09:08 PM UTC

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions