Test failure baseservices/exceptions/simple/ParallelCrashMainThread/ParallelCrashMainThread.sh

[v-wenyuxu]

Run: runtime-coreclr outerloop 20230107.2

Failed test:

coreclr OSX x64 Checked no_tiered_compilation @ OSX.1200.Amd64.Open

- baseservices/exceptions/simple/ParallelCrashMainThread/ParallelCrashMainThread.sh
- baseservices/exceptions/simple/ParallelCrash/ParallelCrash.sh

Error message:

Process terminated. Parallel crash in main thread
at System.Environment.FailFast(System.String)
at ParallelCrash.Main(System.String[])
[createdump] Invalid process id: task_for_pid(99238) FAILED (os/kern) failure (5)
[createdump] This failure may be because createdump or the application is not properly signed and entitled.
[createdump] Failure took 0ms
[createdump] waitpid() returned successfully (wstatus 0000ff00)

cmdLine:/private/tmp/helix/working/B6BE098F/w/B7D109CE/e/baseservices/exceptions/simple/ParallelCrashMainThread/ParallelCrashMainThread.sh Timed Out (timeout in milliseconds: 600000 from variable __TestTimeout, start: 1/7/2023 6:00:33 PM, end: 1/7/2023 6:10:34 PM)

Return code:      -100
Raw output file:      /tmp/helix/working/B6BE098F/w/B7D109CE/uploads/Reports/baseservices.exceptions/simple/ParallelCrashMainThread/ParallelCrashMainThread.output.txt
Raw output:
BEGIN EXECUTION
export CDPATH=/private/tmp/helix/working/B6BE098F/w/B7D109CE/e/baseservices/exceptions/simple/ParallelCrashMainThread
/usr/bin/env bash ../ParallelCrash/ParallelCrash.sh '1'
BEGIN EXECUTION
/tmp/helix/working/B6BE098F/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false ParallelCrash.dll '1'
[createdump] Target process is alive

cmdLine:/private/tmp/helix/working/B6BE098F/w/B7D109CE/e/baseservices/exceptions/simple/ParallelCrashMainThread/ParallelCrashMainThread.sh Timed Out (timeout in milliseconds: 600000 from variable __TestTimeout, start: 1/7/2023 6:00:33 PM, end: 1/7/2023 6:10:34 PM)
Test Harness Exitcode is : -100
To run the test:

set CORE_ROOT=/tmp/helix/working/B6BE098F/p
/private/tmp/helix/working/B6BE098F/w/B7D109CE/e/baseservices/exceptions/simple/ParallelCrashMainThread/ParallelCrashMainThread.sh
Expected: True
Actual:   False


Stack trace
   at baseservices_exceptions._simple_ParallelCraMainThread_ParallelCraMainThread_._simple_ParallelCrashMainThread_ParallelCrashMainThread_sh()
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

Queued	OS	Arch	Column1
2023-01-07T22:51:52.802Z	osx.1200.amd64.open	x64	runtime-coreclr outerloop Checked-no_tiered_compilation
2023-01-07T11:21:08.126Z	osx.1200.amd64.open	x64	runtime-coreclr outerloop Checked-no_tiered_compilation
2023-01-07T22:51:52.802Z	osx.1200.amd64.open	x64	runtime-coreclr outerloop Checked-no_tiered_compilation

Known Issue Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[v-wenyuxu]

Run: runtime-coreclr gcstress-extra 20230204.1

Failed test:

coreclr linux arm Checked gcstress0xc_zapdisable_jitstress2 @ (Ubuntu.1804.Arm32.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7

- baseservices/exceptions/simple/ParallelCrash/ParallelCrash.sh

coreclr linux arm Checked gcstress0xc_zapdisable @ (Ubuntu.1804.Arm32.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7

- baseservices/exceptions/simple/ParallelCrashMainThread/ParallelCrashMainThread.sh

coreclr linux arm64 Checked gcstress0xc_zapdisable_jitstress2 @ (Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8

- baseservices/exceptions/simple/ParallelCrashMainThread/ParallelCrashMainThread.sh

coreclr linux x64 Checked gcstress0xc_jitstress1 @ Ubuntu.1804.Amd64.Open

- baseservices/exceptions/simple/ParallelCrash/ParallelCrash.sh
- baseservices/exceptions/simple/ParallelCrashMainThread/ParallelCrashMainThread.sh

Error message:

Process terminated. Parallel crash in main thread
at System.Environment.FailFast(System.String)
at ParallelCrash.Main(System.String[])
[createdump] waitpid() returned successfully (wstatus 00000000)
/root/helix/work/workitem/e/baseservices/exceptions/simple/ParallelCrash/ParallelCrash.sh: line 425:  5533 Illegal instruction     (core dumped) $LAUNCHER $ExePath "${CLRTestExecutionArguments[@]}"

Return code:      1
Raw output file:      /root/helix/work/workitem/uploads/Reports/baseservices.exceptions/simple/ParallelCrash/ParallelCrash.output.txt
Raw output:
BEGIN EXECUTION
/root/helix/work/correlation/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false ParallelCrash.dll ''
[createdump] Gathering state for process 5533 corerun
[createdump] Crashing thread 0000159d signal 00000006
[createdump] Writing crash report to file /home/helixbot/dotnetbuild/dumps/coredump.5533.dmp.crashreport.json
[createdump] Crash report successfully written
[createdump] Writing minidump with heap to file /home/helixbot/dotnetbuild/dumps/coredump.5533.dmp
[createdump] Written 143884288 bytes (35128 pages) to core file
[createdump] Target process is alive
[createdump] Dump successfully written in 763ms
Expected: 134
Actual: 132
END EXECUTION - FAILED
Test failed. Trying to see if dump file was created in /home/helixbot/dotnetbuild/dumps since 2/5/2023 12:33:51 AM
Processing /home/helixbot/dotnetbuild/dumps/coredump.5533.dmp.crashreport.json
Printing stacktrace from '/home/helixbot/dotnetbuild/dumps/coredump.5533.dmp.crashreport.json'
Invoking llvm-symbolizer --pretty-print
Errors while running llvm-symbolizer --pretty-print
System.ComponentModel.Win32Exception (2): An error occurred trying to start process 'llvm-symbolizer' with working directory '/root/helix/work/workitem/e'. No such file or directory
at System.Diagnostics.Process.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, String[] envp, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean usesTerminal, Boolean throwOnNoExec)
at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)
at CoreclrTestLib.CoreclrTestWrapperLib.TryPrintStackTraceFromCrashReport(String crashReportJsonFile, StreamWriter outputWriter)
Finish looking for *.crashreport.json. No new files created.
Test Harness Exitcode is : 1
To run the test:

set CORE_ROOT=/root/helix/work/correlation
/root/helix/work/workitem/e/baseservices/exceptions/simple/ParallelCrash/ParallelCrash.sh
Expected: True
Actual:   False


Stack trace
   at baseservices_exceptions._simple_ParallelCra_ParallelCra_._simple_ParallelCrash_ParallelCrash_sh()
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

[v-wenyuxu]

Failed in Run: runtime-coreclr gcstress-extra 20230212.1

Failed tests:

coreclr linux x64 Checked gcstress0xc_jitstress2 @ Ubuntu.1804.Amd64.Open
    - baseservices/exceptions/simple/ParallelCrash/ParallelCrash.sh
coreclr linux x64 Checked gcstress0xc_tailcallstress @ Ubuntu.1804.Amd64.Open
    - baseservices/exceptions/simple/ParallelCrash/ParallelCrash.sh 

Error message:

Process terminated. Parallel crash in main thread
  at System.Environment.FailFast(System.String)
  at ParallelCrash.Main(System.String[])
[createdump] waitpid() returned successfully (wstatus 00000000)
/datadisks/disk1/work/9E9308D7/w/C40B09F9/e/baseservices/exceptions/simple/ParallelCrash/ParallelCrash.sh: line 425: 13244 Segmentation fault      (core dumped) $LAUNCHER $ExePath "${CLRTestExecutionArguments[@]}"

Return code:      1
Raw output file:      /datadisks/disk1/work/9E9308D7/w/C40B09F9/uploads/Reports/baseservices.exceptions/simple/ParallelCrash/ParallelCrash.output.txt
Raw output:
BEGIN EXECUTION
/datadisks/disk1/work/9E9308D7/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false ParallelCrash.dll ''
[createdump] Gathering state for process 13244 corerun
[createdump] Crashing thread 000033bc signal 00000006
[createdump] Writing crash report to file /home/helixbot/dotnetbuild/dumps/coredump.13244.dmp.crashreport.json
[createdump] Crash report successfully written
[createdump] Writing minidump with heap to file /home/helixbot/dotnetbuild/dumps/coredump.13244.dmp
[createdump] Written 145395712 bytes (35497 pages) to core file
[createdump] Target process is alive
[createdump] Dump successfully written in 597ms
Expected: 134
Actual: 139
END EXECUTION - FAILED
Test failed. Trying to see if dump file was created in /home/helixbot/dotnetbuild/dumps since 2/12/2023 10:50:58 PM
Processing /home/helixbot/dotnetbuild/dumps/coredump.13244.dmp.crashreport.json
Printing stacktrace from '/home/helixbot/dotnetbuild/dumps/coredump.13244.dmp.crashreport.json'
Invoking llvm-symbolizer --pretty-print
Stack trace:
----------------------------------
Thread Id: 0x33bc
     Child SP               IP Call Site
0x7ffc38633a10 0x7f4b6e20432a libpthread.so.0!waitpid
0x7ffc38633a40 0x7f4b6d3d0560 libcoreclr.so!PROCCreateCrashDump(std::vector<char const*, std::allocator<char const*> >&, char*, int) at /__w/1/s/src/coreclr/pal/src/thread/process.cpp:2223:22
0x7ffc38633ab0 0x7f4b6d3d1304 libcoreclr.so!PROCCreateCrashDumpIfEnabled at /__w/1/s/src/coreclr/pal/src/thread/process.cpp:2413:9
0x7ffc38633b40 0x7f4b6d3cdf21 libcoreclr.so!PROCAbort at /__w/1/s/src/coreclr/pal/src/thread/process.cpp:2441:5
0x7ffc38633b60 0x7f4b6d3cdcbf libcoreclr.so!PROCEndProcess(void*, unsigned int, int) at /__w/1/s/src/coreclr/pal/src/thread/process.cpp:1346:13
0x7ffc38633b90 0x7f4b6d3cdd55 libcoreclr.so!TerminateProcess at /__w/1/s/src/coreclr/pal/src/thread/process.cpp:1239:11
0x7ffc38633bc0 0x7f4b6cf7c220 libcoreclr.so!SafeExitProcess(unsigned int, ShutdownCompleteAction) at /__w/1/s/src/coreclr/vm/eepolicy.cpp:0:9
0x7ffc38633c00 0x7f4b6cf7d53b libcoreclr.so!CHECK::CHECK() at /__w/1/s/src/coreclr/inc/check.h:115:17
(inlined by) EEPolicy::HandleFatalError(unsigned int, unsigned long, char16_t const*, _EXCEPTION_POINTERS*, char16_t const*, char16_t const*) at /__w/1/s/src/coreclr/vm/eepolicy.cpp:795:5
0x7ffc386341f0 0x7f4b6d2ce3c7 libcoreclr.so!SBuffer::~SBuffer() at /__w/1/s/src/coreclr/inc/sbuffer.inl:180:9
(inlined by) SystemNative::GenericFailFast(REF<StringObject>, REF<ExceptionObject>, unsigned long, unsigned int, REF<StringObject>) at /__w/1/s/src/coreclr/classlibnative/bcltype/system.cpp:289:5
0x7ffc386344d0 0x7f4b6d2ce6ad libcoreclr.so!HelperMethodFrame_1OBJ::Pop() at /__w/1/s/src/coreclr/classlibnative/../vm/frames.h:1421:28
(inlined by) FrameWithCookie<HelperMethodFrame_1OBJ>::Pop() at /__w/1/s/src/coreclr/classlibnative/../vm/frames.h:3249:47
(inlined by) SystemNative::FailFast(StringObject*) at /__w/1/s/src/coreclr/classlibnative/bcltype/system.cpp:308:5
0x7ffc386346a0 0x7f4aeeee1b3b ParallelCrash.dll!ParallelCrash.Main(System.String[])
----------------------------------
Thread Id: 0x33bd
     Child SP               IP Call Site
0x7f4b6c8e6d60 0x7f4b6d5d6bb9 libc.so.6!__poll
0x7f4b6c8e6d90 0x7f4b6d3be83e libcoreclr.so!CorUnix::CPalSynchronizationManager::ThreadPrepareForShutdown() at /__w/1/s/src/coreclr/pal/src/sync

Stack trace:

  at baseservices_exceptions._simple_ParallelCra_ParallelCra_._simple_ParallelCrash_ParallelCrash_sh()
  at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
  at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

[BruceForstall]

Some recent failures of ParallelCrash:

Queued	Pipeline	Pipeline_Configuration	OS	Arch
2023-02-13T07:55:46.964Z	runtime-coreclr crossgen2-composite gcstress	Checked-gcstress0xc_jitstress2	ubuntu.1804.amd64.open.rt	x64
2023-02-13T03:39:46.694Z	runtime-coreclr r2r-extra	Checked-gcstress0xc	ubuntu.1804.amd64.open.rt	x64
2023-02-12T22:39:43.294Z	runtime-coreclr gcstress-extra	Checked-gcstress0xc_tailcallstress	ubuntu.1804.amd64.open.rt	x64
2023-02-12T22:39:42.157Z	runtime-coreclr gcstress-extra	Checked-gcstress0xc_jitstress2	ubuntu.1804.amd64.open.rt	x64
2023-02-12T07:41:18.287Z	runtime-coreclr crossgen2-composite gcstress	Checked-gcstress0xc_zapdisable_heapverify1	ubuntu.1804.amd64.open.rt	x64
2023-02-11T22:39:54.718Z	runtime-coreclr gcstress-extra	Checked-gcstress0xc_jitstress1	ubuntu.1804.amd64.open.rt	x64
2023-02-06T08:23:41.332Z	runtime-coreclr crossgen2-composite gcstress	Checked-gcstress0xc_jitstress1	osx.1200.arm64.open	arm64
2023-02-06T08:00:30.958Z	runtime-coreclr crossgen2-composite gcstress	Checked-gcstress0xc_jitstress2	ubuntu.1804.amd64.open.rt	x64
2023-02-06T08:00:28.275Z	runtime-coreclr crossgen2-composite gcstress	Checked-gcstress0xc_zapdisable	ubuntu.1804.amd64.open.rt	x64
2023-02-05T22:51:16.936Z	runtime-coreclr gcstress-extra	Checked-gcstress0xc_tailcallstress	ubuntu.1804.armarch.open	arm64
2023-02-05T07:51:45.652Z	runtime-coreclr crossgen2-composite gcstress	Checked-gcstress0xc_jitstress1	ubuntu.1804.amd64.open.rt	x64
2023-02-05T07:51:40.417Z	runtime-coreclr crossgen2-composite gcstress	Checked-gcstress0xc_jitminopts_heapverify1	ubuntu.1804.amd64.open.rt	x64
2023-02-05T07:51:40.417Z	runtime-coreclr crossgen2-composite gcstress	Checked-gcstress0xc_jitminopts_heapverify1	ubuntu.1804.amd64.open.rt	x64
2023-02-04T22:57:30.335Z	runtime-coreclr gcstress-extra	Checked-gcstress0xc_jitstress1	ubuntu.1804.amd64.open.rt	x64
2023-02-04T22:56:11.956Z	runtime-coreclr gcstress-extra	Checked-gcstress0xc_zapdisable_jitstress2	ubuntu.1804.armarch.open	arm
2023-01-23T08:15:21.364Z	runtime-coreclr crossgen2-composite gcstress	Checked-gcstress0xc_zapdisable	ubuntu.1804.amd64.open.rt	x64
2023-01-23T08:15:20.208Z	runtime-coreclr crossgen2-composite gcstress	Checked-gcstress0xc_jitstress1	ubuntu.1804.amd64.open.rt	x64
2023-01-22T22:36:00.945Z	runtime-coreclr gcstress-extra	Checked-gcstress0xc_jitstress1	ubuntu.1804.amd64.open.rt	x64
2023-01-22T08:03:24.673Z	runtime-coreclr crossgen2-composite gcstress	Checked-gcstress0xc_jitstress2	ubuntu.1804.amd64.open.rt	x64
2023-01-21T22:40:06.836Z	runtime-coreclr gcstress-extra	Checked-gcstress0xc_zapdisable	ubuntu.1804.armarch.open	arm64
2023-01-21T22:40:04.914Z	runtime-coreclr gcstress-extra	Checked-gcstress0xc_tailcallstress	ubuntu.1804.armarch.open	arm64
2023-01-07T22:51:52.802Z	runtime-coreclr outerloop	Checked-no_tiered_compilation	osx.1200.amd64.open	x64

[BruceForstall]

Since all recent failures have been GCStress, I'm marking it as such. The OSX failure hasn't happened in a while.

[BruceForstall]

Also, the runtime-coreclr outerloop pipeline hasn't failed, so changing the blocking label