Tests timing out with incomplete log

[safern]

  Discovering: System.Runtime.Extensions.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Runtime.Extensions.Tests (found 837 of 870 test cases)
  Starting:    System.Runtime.Extensions.Tests (parallel test collections = on, max threads = 2)
    System.Tests.EnvironmentTests.GetFolderPath_UWP_NotEmpty [SKIP]
      Condition(s) not met: "IsWindows10Version1709OrGreater", "IsInAppContainer"
    System.Tests.EnvironmentTests.GetFolderPath_UWP_ExistAndAccessible [SKIP]
      Condition(s) not met: "IsWindows10Version1709OrGreater", "IsInAppContainer"
    System.Tests.EnvironmentTests.UserInteractive_WindowsNano [SKIP]
      Condition(s) not met: "IsWindowsNanoServer"
   System.Runtime.Extensions.Tests: [Long Running Test] 'System.Tests.Environment_Exit.ExitCode_VoidMainAppReturnsSetValue', Elapsed: 00:02:28
   System.Runtime.Extensions.Tests: [Long Running Test] 'System.Tests.Environment_Exit.ExitCode_VoidMainAppReturnsSetValue', Elapsed: 00:04:28
   System.Runtime.Extensions.Tests: [Long Running Test] 'System.Tests.Environment_Exit.ExitCode_VoidMainAppReturnsSetValue', Elapsed: 00:06:28
   System.Runtime.Extensions.Tests: [Long Running Test] 'System.Tests.Environment_Exit.ExitCode_VoidMainAppReturnsSetValue', Elapsed: 00:08:28
   System.Runtime.Extensions.Tests: [Long Running Test] 'System.Tests.Environment_Exit.ExitCode_VoidMainAppReturnsSetValue', Elapsed: 00:10:28
   System.Runtime.Extensions.Tests: [Long Running Test] 'System.Tests.Environment_Exit.ExitCode_VoidMainAppReturnsSetValue', Elapsed: 00:12:28
   System.Runtime.Extensions.Tests: [Long Running Test] 'System.Tests.Environment_Exit.ExitCode_VoidMainAppReturnsSetValue', Elapsed: 00:14:28

https://dev.azure.com/dnceng/public/_build/results?buildId=627022&view=ms.vss-test-web.build-test-results-tab&runId=19596708&resultId=183795&paneView=debug

https://helix.dot.net/api/2019-06-17/jobs/6bdbf48d-5deb-436c-8f91-8f96823af464/workitems/System.Runtime.Extensions.Tests/console

Failed in:
#35169

Configuration: netcoreapp5.0-Windows_NT-Debug-x64-CoreCLR_release-Windows.7.Amd64

cc: @jkotas @stephentoub

[jkotas]

The log is incomplete. This is likely combination of #35451 and infrastructure problem (machine getting recycled or something similar).

Tagging subscribers to this area: @safern, @ViktorHofer
Notify danmosemsft if you want to be subscribed.

[safern]

I think when the work item exceeds the helix timeout which is 15 mins helix kills the process, that’s why the log is incomplete, but @MattGal can confirm.

[MattGal]

Yes. This is a timeout. I've had the conversation several times recently, and after investigation we determined that we would have to change the Helix API to make this more obvious, as adding a new enum would break clients.

If you just delete "/console" from that URI you get the other log:
https://helix.dot.net/api/2019-06-17/jobs/6bdbf48d-5deb-436c-8f91-8f96823af464/workitems/System.Runtime.Extensions.Tests --> this log

... which contains:

2020-05-02T01:27:10.589Z	INFO   	servicebusrepository(109)	renew_workitem_lock	Renewed work item lock. Status Code: 200
2020-05-02T01:27:53.402Z	INFO   	job(39)	kill	Begin killing timed-out process(es)
2020-05-02T01:27:53.402Z	ERROR  	job(48)	kill	Job running for too long. Killing...
2020-05-02T01:27:53.402Z	INFO   	job(54)	kill	Finished killing timed-out process(es)
2020-05-02T01:27:53.402Z	ERROR  	executor(652)	_execute_command	Executor timed out after 900 seconds and was killed.
2020-05-02T01:27:53.402Z	INFO   	event(42)	send	Sending event type WorkItemTimeout
2020-05-02T01:27:53.589Z	INFO   	saferequests(87)	request_with_retry	Response complete with status code '201'
2020-05-02T01:27:53.589Z	INFO   	executor(618)	_dump_file_upload	No dump files to upload
2020-05-02T01:27:53.589Z	INFO   	executor(674)	_execute_command	Finished _execute_command, exit code: -3
2020-05-02T01:27:53.589Z	INFO   	logs(153)	__exit__	Uploading self-uploading log file C:\h\logs\24c41cd4-1b79-41c4-9a13-e6dc6c0b89c8.log (Client: a002EXQ)

I sympathize with how this isn't handing you the information you need as directly as you'd like, but if you fix your hangs / slow tests and/or increase your timeouts this will go away.

[jkotas]

Well, we cannot fix our hangs unless we know where they are. We need process dumps taken when things are killed due to timeouts like this.