dotnet test on linux container randomly crashing host

[ealsur]

Description

We have an Azure Pipeline that runs dotnet test on the solution. The solution contains several test projects. Non-deterministically, we get Testhost aborted failures, across different test projects, sometimes none, for the same source code.

This using .NET 8.0.408 on Linux.

Reproduction Steps

When the host crashes, we see these logs using --diag

TpTrace Verbose: 0 : 1307, 10, 2025/05/21, 18:27:09.172, 249491535604, vstest.console.dll, TestHostManagerCallbacks.StandardOutputReceivedCallback Test host standard output line: 
TpTrace Warning: 0 : 1307, 11, 2025/05/21, 18:27:09.172, 249491562318, vstest.console.dll, TestHostManagerCallbacks.ErrorReceivedCallback Test host standard error line: 
TpTrace Verbose: 0 : 1307, 11, 2025/05/21, 18:27:09.175, 249494786437, vstest.console.dll, TestHostProvider.ExitCallBack: Host exited starting callback.
TpTrace Error: 0 : 1307, 11, 2025/05/21, 18:27:09.175, 249494945026, vstest.console.dll, TestHostManagerCallbacks.ExitCallBack: Testhost processId: 1918 exited with exitcode: 139 error: ''
TpTrace Verbose: 0 : 1307, 11, 2025/05/21, 18:27:09.175, 249495007524, vstest.console.dll, DotnetTestHostManager.OnHostExited: invoking OnHostExited callback
TpTrace Verbose: 0 : 1307, 11, 2025/05/21, 18:27:09.175, 249495079620, vstest.console.dll, CrossPlatEngine.TestHostManagerHostExited: calling on client process exit callback.
TpTrace Information: 0 : 1307, 11, 2025/05/21, 18:27:09.175, 249495154571, vstest.console.dll, TestRequestSender.OnClientProcessExit: Test host process exited. Standard error: 
TpTrace Verbose: 0 : 1307, 5, 2025/05/21, 18:27:09.176, 249495239770, vstest.console.dll, TestRequestSender.WaitForRequestHandlerConnection: waiting took 1331 ms, with timeout 90000 ms, and result 2, which is failure.
TpTrace Information: 0 : 1307, 11, 2025/05/21, 18:27:09.176, 249495242216, vstest.console.dll, SocketServer.Stop: Stop server endPoint: 127.0.0.1:34723
TpTrace Verbose: 0 : 1307, 5, 2025/05/21, 18:27:09.176, 249495307237, vstest.console.dll, Test host failed to start Test host launched:True test host exited: True
TpTrace Information: 0 : 1307, 11, 2025/05/21, 18:27:09.176, 249495373343, vstest.console.dll, SocketServer.Stop: Cancellation requested. Stopping message loop.
TpTrace Verbose: 0 : 1307, 11, 2025/05/21, 18:27:09.176, 249495618365, vstest.console.dll, MulticastDelegateUtilities.SafeInvoke: HostProviderEvents.OnHostExited: Invoking callback 1/1 for Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyOperationManager., took 0 ms.
TpTrace Error: 0 : 1307, 5, 2025/05/21, 18:27:09.177, 249496714440, vstest.console.dll, ProxyExecutionManager.StartTestRun: Failed to start test run: Microsoft.VisualStudio.TestPlatform.ObjectModel.TestPlatformException: Testhost process for source(s) '/__w/1/s/<obfuscatedproject>.dll' exited with error: . Please check the diagnostic logs for more information.
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyOperationManager.ThrowOnTestHostExited(IEnumerable`1 sources, Boolean testHostExited) in /_/src/Microsoft.TestPlatform.CrossPlatEngine/Client/ProxyOperationManager.cs:line 520
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyOperationManager.SetupChannel(IEnumerable`1 sources, String runSettings) in /_/src/Microsoft.TestPlatform.CrossPlatEngine/Client/ProxyOperationManager.cs:line 290
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyExecutionManager.InitializeTestRun(TestRunCriteria testRunCriteria, IInternalTestRunEventsHandler eventHandler) in /_/src/Microsoft.TestPlatform.CrossPlatEngine/Client/ProxyExecutionManager.cs:line 182
TpTrace Verbose: 0 : 1307, 5, 2025/05/21, 18:27:09.179, 249498436005, vstest.console.dll, TestRunRequest:SendTestRunMessage: Starting.
TpTrace Verbose: 0 : 1307, 5, 2025/05/21, 18:27:09.179, 249498630291, vstest.console.dll, MulticastDelegateUtilities.SafeInvoke: TestRun.LogMessages: Invoking callback 1/1 for Microsoft.VisualStudio.TestPlatform.CommandLine.TestRunResultAggregator., took 0 ms.
TpTrace Verbose: 0 : 1307, 4, 2025/05/21, 18:27:09.179, 249498684596, vstest.console.dll, MulticastDelegateUtilities.SafeInvoke: InternalTestLoggerEvents.SendTestRunMessage: Invoking callback 1/2 for Microsoft.VisualStudio.TestPlatform.Extensions.TrxLogger.TrxLogger., took 0 ms.
TpTrace Information: 0 : 1307, 5, 2025/05/21, 18:27:09.179, 249498757541, vstest.console.dll, TestRunRequest:SendTestRunMessage: Completed.
TpTrace Verbose: 0 : 1307, 4, 2025/05/21, 18:27:09.179, 249498865366, vstest.console.dll, MulticastDelegateUtilities.SafeInvoke: InternalTestLoggerEvents.SendTestRunMessage: Invoking callback 2/2 for Microsoft.VisualStudio.TestPlatform.CommandLine.Internal.ConsoleLogger., took 0 ms.
TpTrace Verbose: 0 : 1307, 5, 2025/05/21, 18:27:09.189, 249508779204, vstest.console.dll, TestRequestSender.EndSession: Sending end session.
TpTrace Verbose: 0 : 1307, 5, 2025/05/21, 18:27:09.189, 249508869554, vstest.console.dll, ProxyOperationManager.Close: waiting for test host to exit for 100 ms
TpTrace Information: 0 : 1307, 5, 2025/05/21, 18:27:09.189, 249508962930, vstest.console.dll, SocketServer.Stop: Stop server endPoint: 127.0.0.1:34723
TpTrace Information: 0 : 1307, 5, 2025/05/21, 18:27:09.189, 249509011522, vstest.console.dll, SocketServer.Stop: Cancellation requested. Stopping message loop.
TpTrace Information: 0 : 1307, 5, 2025/05/21, 18:27:09.189, 249509077336, vstest.console.dll, Closing the connection
TpTrace Verbose: 0 : 1307, 5, 2025/05/21, 18:27:09.194, 249513810669, vstest.console.dll, ParallelProxyExecutionManager: HandlePartialRunComplete: Total workloads = 1, Total started clients = 1 Total completed clients = 1, Run complete = True, Run canceled: False.
TpTrace Verbose: 0 : 1307, 5, 2025/05/21, 18:27:09.194, 249513878186, vstest.console.dll, ParallelProxyExecutionManager: HandlePartialRunComplete: All runs completed stopping all managers.
TpTrace Verbose: 0 : 1307, 5, 2025/05/21, 18:27:09.194, 249513943199, vstest.console.dll, ParallelOperationManager.StopAllManagers: Stopping all managers.
TpTrace Verbose: 0 : 1307, 5, 2025/05/21, 18:27:09.194, 249514001689, vstest.console.dll, ParallelOperationManager.ClearSlots: Clearing all slots. Slots should accept more work: False
TpTrace Verbose: 0 : 1307, 5, 2025/05/21, 18:27:09.194, 249514078955, vstest.console.dll, ParallelOperationManager.SetOccupiedSlotCount: Setting slot counts AvailableSlotCount = 1, OccupiedSlotCount = 0.
TpTrace Verbose: 0 : 1307, 5, 2025/05/21, 18:27:09.194, 249514148716, vstest.console.dll, Occupied slots:

TpTrace Information: 0 : 1307, 4, 2025/05/21, 18:27:09.226, 249545245008, vstest.console.dll, Converter.ToCollectionEntries: Converting attachmentSets  to collection entries.
TpTrace Information: 0 : 1307, 4, 2025/05/21, 18:27:09.229, 249548424212, vstest.console.dll, Results File: /__w/1/TestResults/_487e1012aa75_2025-05-21_18_27_09.trx
TpTrace Verbose: 0 : 1307, 4, 2025/05/21, 18:27:09.229, 249548494765, vstest.console.dll, MulticastDelegateUtilities.SafeInvoke: InternalTestLoggerEvents.SendTestRunComplete: Invoking callback 1/3 for Microsoft.VisualStudio.TestPlatform.Extensions.TrxLogger.TrxLogger., took 15 ms.
TpTrace Verbose: 0 : 1307, 4, 2025/05/21, 18:27:09.234, 249553203572, vstest.console.dll, MulticastDelegateUtilities.SafeInvoke: InternalTestLoggerEvents.SendTestRunComplete: Invoking callback 2/3 for Microsoft.VisualStudio.TestPlatform.CommandLine.Internal.ConsoleLogger., took 4 ms.
TpTrace Verbose: 0 : 1307, 4, 2025/05/21, 18:27:09.234, 249553569952, vstest.console.dll, MulticastDelegateUtilities.SafeInvoke: InternalTestLoggerEvents.SendTestRunComplete: Invoking callback 3/3 for Microsoft.TestPlatform.Extensions.BlameDataCollector.BlameLogger., took 0 ms.
TpTrace Verbose: 0 : 1307, 5, 2025/05/21, 18:27:09.234, 249553839931, vstest.console.dll, TestRunRequest:TestRunComplete: Starting. IsAborted:True IsCanceled:False.
TpTrace Verbose: 0 : 1307, 5, 2025/05/21, 18:27:09.234, 249553997888, vstest.console.dll, ParallelOperationManager.DoActionOnAllManagers: Calling an action on all managers.
TpTrace Verbose: 0 : 1307, 5, 2025/05/21, 18:27:09.235, 249554950595, vstest.console.dll, TestLoggerManager.HandleTestRunComplete: Ignoring as the object is disposed.
TpTrace Verbose: 0 : 1307, 5, 2025/05/21, 18:27:09.235, 249555017581, vstest.console.dll, MulticastDelegateUtilities.SafeInvoke: TestRun.TestRunComplete: Invoking callback 1/2 for Microsoft.VisualStudio.TestPlatform.CommandLine.TestRunResultAggregator., took 0 ms.
TpTrace Verbose: 0 : 1307, 5, 2025/05/21, 18:27:09.235, 249555111137, vstest.console.dll, MulticastDelegateUtilities.SafeInvoke: TestRun.TestRunComplete: Invoking callback 2/2 for Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.RunTestsArgumentExecutor+TestRunRequestEventsRegistrar., took 0 ms.
TpTrace Information: 0 : 1307, 5, 2025/05/21, 18:27:09.236, 249555204132, vstest.console.dll, TestRunRequest:TestRunComplete: Completed.
TpTrace Verbose: 0 : 1307, 1, 2025/05/21, 18:27:09.236, 249555232979, vstest.console.dll, TestRunRequest.Dispose: Starting.
TpTrace Verbose: 0 : 1307, 5, 2025/05/21, 18:27:09.236, 249555253205, vstest.console.dll, ParallelProxyExecutionManager.StartTestRunOnConcurrentManager: Execution starting. Started clients: 1
TpTrace Information: 0 : 1307, 1, 2025/05/21, 18:27:09.236, 249555307268, vstest.console.dll, TestRunRequest.Dispose: Completed.
TpTrace Information: 0 : 1307, 1, 2025/05/21, 18:27:09.236, 249555387900, vstest.console.dll, TestRequestManager.RunTests: run tests completed.
TpTrace Information: 0 : 1307, 1, 2025/05/21, 18:27:09.236, 249555469164, vstest.console.dll, RunTestsArgumentProcessor:Execute: Test run is completed.
TpTrace Verbose: 0 : 1307, 5, 2025/05/21, 18:27:09.236, 249555487467, vstest.console.dll, TestRunRequest:SendTestRunMessage: Starting.
TpTrace Warning: 0 : 1307, 5, 2025/05/21, 18:27:09.236, 249555565944, vstest.console.dll, TestRunRequest.SendTestRunMessage: Ignoring as the object is disposed.
TpTrace Verbose: 0 : 1307, 5, 2025/05/21, 18:27:09.236, 249555662366, vstest.console.dll, ProxyExecutionManager: Test host is always Lazy initialize.
TpTrace Verbose: 0 : 1307, 1, 2025/05/21, 18:27:09.236, 249555724686, vstest.console.dll, Executor.Execute: Exiting with exit code of 1
TpTrace Verbose: 0 : 1307, 5, 2025/05/21, 18:27:09.236, 249555743950, vstest.console.dll, TestRequestSender.InitializeCommunication: initialize communication. 
TpTrace Information: 0 : 1307, 5, 2025/05/21, 18:27:09.236, 249555960598, vstest.console.dll, SocketServer.Start: Listening on endpoint : 127.0.0.1:43347
TpTrace Verbose: 0 : 1307, 5, 2025/05/21, 18:27:09.237, 249556255264, vstest.console.dll, DotnetTestHostmanager.GetTestHostProcessStartInfo: Platform environment 'X64' target architecture 'X64' framework '.NETCoreApp,Version=v8.0' OS 'Unix'
TpTrace Verbose: 0 : 1307, 5, 2025/05/21, 18:27:09.237, 249556323092, vstest.console.dll, DotnetTestHostmanager: Adding  --runtimeconfig "/__w/1/s/<obfuscatedproject>.runtimeconfig.json" in args
TpTrace Verbose: 0 : 1307, 5, 2025/05/21, 18:27:09.237, 249556365582, vstest.console.dll, DotnetTestHostmanager: Adding  --depsfile "/__w/1/s/<obfuscatedproject>.deps.json" in args
lticastDelegateUtilities.SafeInvoke: InternalTestLoggerEvents.SendTestResult: Invoking callback 2/2 for Microsoft.VisualStudio.TestPlatform.CommandLine.Internal.ConsoleLogger., took 0 ms.

The crash seems to always be with errorcode: 139 but there is no information at all on what is the cause.

Adding --blame --blame-crash --blame-crash-collect-always to the CLI but we see no dumps.

On the dotnet test output we see:

Data collector 'Blame' message: System.IO.FileNotFoundException: Collect dump was enabled but no dump file was generated.
   at Microsoft.TestPlatform.Extensions.BlameDataCollector.ProcessDumpUtility.GetDumpFiles(Boolean warnOnNoDumpFiles, Boolean processCrashed) in /_/src/Microsoft.TestPlatform.Extensions.BlameDataCollector/ProcessDumpUtility.cs:line 84
   at Microsoft.TestPlatform.Extensions.BlameDataCollector.BlameCollector.SessionEndedHandler(Object sender, SessionEndEventArgs args) in /_/src/Microsoft.TestPlatform.Extensions.BlameDataCollector/BlameCollector.cs:line 534.

Expected behavior

Some form of explanation of what is causing the crash. Attachments contain dumps.

Actual behavior

No indication of what is the reason for the crash. No dumps in Attachments.

Regression?

No response

Known Workarounds

No response

Configuration

• .NET 8.0.408
• mcr.microsoft.com/onebranch/azurelinux/build:3.0
• x64
• It doesn't repro locally

Other information

No response

[ealsur]

Looking at the log.datacollector.. files, I see this:

TpTrace Information: 0 : 1538, 1, 2025/05/21, 18:27:04.353, 244672232886, datacollector.dll, DataCollectionRequestHandler.ProcessRequests : DataCollection started.
TpTrace Information: 0 : 1538, 1, 2025/05/21, 18:27:04.373, 244692488147, datacollector.dll, DataCollectionRequestHandler.ProcessRequests: Datacollector received message: (DataCollection.TestHostLaunched) -> {"MessageType":"DataCollection.TestHostLaunched","Payload":{"ProcessId":1653}}
TpTrace Information: 0 : 1538, 1, 2025/05/21, 18:27:04.375, 244694516608, datacollector.dll, DynamicCoverageDataCollector.TestHostLaunched: TestHostLaunched event received: ProcessId: 1653
TpTrace Verbose: 0 : 1538, 1, 2025/05/21, 18:27:04.375, 244694835846, datacollector.dll, MulticastDelegateUtilities.SafeInvoke: DataCollectionEvents.TestHostLaunched: Invoking callback 1/2 for Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollectorWithAttachmentProcessorAndTelemetry., took 0 ms.
TpTrace Information: 0 : 1538, 1, 2025/05/21, 18:27:04.379, 244698727824, datacollector.dll, ProcessDumpUtility.CrashDump: Creating a crash dumper for process dotnet (1653). If crash happens, dumper will try to create 'Full' dump and store it temporarily in path '/__w/_temp/80dc43fe-e0cb-48c5-bf94-461841a82ce7'. Later dumps will become attachments and will be moved to TestResults directory.
TpTrace Information: 0 : 1538, 1, 2025/05/21, 18:27:04.379, 244699120802, datacollector.dll, CrashDumperFactory: Creating dumper for Microsoft Azure Linux 3.0 with target framework .NETCoreApp,Version=v8.0.
TpTrace Information: 0 : 1538, 1, 2025/05/21, 18:27:04.380, 244699372912, datacollector.dll, CrashDumperFactory: This is Microsoft Azure Linux 3.0 on .NETCoreApp,Version=v8.0 .NETClient dumper which uses env variables to collect crashdumps of testhost and any child process that is .NET 5 or newer.
TpTrace Verbose: 0 : 1538, 1, 2025/05/21, 18:27:04.382, 244701556438, datacollector.dll, MulticastDelegateUtilities.SafeInvoke: DataCollectionEvents.TestHostLaunched: Invoking callback 2/2 for Microsoft.TestPlatform.Extensions.BlameDataCollector.BlameCollector., took 6 ms.
TpTrace Error: 0 : 1538, 1, 2025/05/21, 18:27:07.253, 247572426347, datacollector.dll, Program.Main: Error occurred during initialization of Datacollector : System.IO.EndOfStreamException: Unable to read beyond the end of the stream.
   at System.IO.BinaryReader.Read7BitEncodedInt()
   at System.IO.BinaryReader.ReadString()
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.SocketCommunicationManager.ReceiveRawMessage() in /_/src/Microsoft.TestPlatform.CommunicationUtilities/SocketCommunicationManager.cs:line 321
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.SocketCommunicationManager.ReceiveMessage() in /_/src/Microsoft.TestPlatform.CommunicationUtilities/SocketCommunicationManager.cs:line 289
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.DataCollection.DataCollectionRequestHandler.ProcessRequests() in /_/src/Microsoft.TestPlatform.CommunicationUtilities/DataCollectionRequestHandler.cs:line 178
   at Microsoft.VisualStudio.TestPlatform.DataCollector.DataCollectorMain.Run(String[] args) in /_/src/datacollector/DataCollectorMain.cs:line 150
   at Microsoft.VisualStudio.TestPlatform.DataCollector.Program.Main(String[] args) in /_/src/datacollector/Program.cs:line 25

[ealsur]

I managed to get ahold of this temporary dump from /__w/_temp/80dc43fe-e0cb-48c5-bf94-461841a82ce7 but it doesn't have any actionable data as far as I can see:

[jeffschwMSFT]

@tkapin does your team own these components? Is there a repro we can move this too?

[tkapin]

Absolutely. @pavelhorak and team - could you please check if this is a VSTest (or related) issue? A repro would help, of course.

[pavelhorak]

@nohwnd is looking into this

[nohwnd]

@ealsur the log above mentions that the testhost is exiting with 139 on linux which is probably means it is being killed by the OS via SIGSEGV

TpTrace Error: 0 : 1307, 11, 2025/05/21, 18:27:09.175, 249494945026, vstest.console.dll, TestHostManagerCallbacks.ExitCallBack: Testhost processId: 1918 exited with exitcode: 139 error: ''

This is more likely to be an error in the code that is being tested that will then kill the whole process (testhost), rather than testhost or runtime itself (because in such cases we get multiple reports of such problems).

Internet recommends to first look at kernel logs to see what could be the cause. Have you tried looking at the dump as native / mixed to see if there are any available stacks?

[ealsur]

@nohwnd Thanks for the insight! Yes, we were on that track and we were trying to get a dump of the crash to understand what code was causing the issue.
But the problem is that using --blame --blame-crash --blame-crash-collect-always is not generating any crash dump. The BlameCollector is crashing while attempting to generate the crash apparently.
On the dotnet test console output we see: Collect dump was enabled but no dump file was generated.
And there are no attachments after the run generated.

[nohwnd]

You won’t be able to make a dump like that. Blame on .NET relies on the .NET built in dumper which won’t create a dump in this case. The app is killed externally by the os, so it won’t get more execution time to make the dump.

[ealsur]

The challenge is this is running in Azure Pipelines, do you know if there is a way to obtain some log that can help identify what is causing the crash? I tried adding logs for unhandled exceptions on the AppDomain, but nothing is appearing.

[nohwnd]

If you will search online on how to debug this kind of problem, there should be a command that prints the system logs, and you can just print that to screen (e.g. via cat) on the azdo agent to get the info from logs. All the highlevel .net constructs won’t work because your process is (most likely) faulting and force killed by os before it can cause more damage.

Maybe someone else from runtime can offer advice on how they capture and solve such issues. @ivanpovazan you go pretty low level, some ideas?:)