slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1272635	dotnet/maui	RunOniOS_BlazorDebug macOS-arm64		dotnet/maui#33749
1272918	maestro-auth-test/helix-queue-insights-test	HelixTests	Log
2890820	dotnet-maui	Controls CollectionView	Log
1272666	dotnet/maui	RunOniOS_MauiRelease macOS	Log
1272640	dotnet/maui	RunOniOS_BlazorRelease macOS	Log	dotnet/maui#32296
1272667	dotnet/maui	Controls Layout	Log
1272642	dotnet/maui	Controls CollectionView	Log
1272660	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log
2890841	dotnet-maui	RunOniOS_MauiNativeAOT macOS	Log
1272398	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log
1271887	dotnet/maui	RunOniOS_MauiRelease macOS	Log
2890817	dotnet-performance	Performance maui_scenarios_ios Interpreter CoreCLR osx 15 x64 iPhoneMini12 main	Log
1272407	dotnet/maui	Build macOS	Log	dotnet/maui#33756
1272408	dotnet/maui	Controls Layout	Log	dotnet/maui#33756
2890746	dotnet-runtime	Performance ios_scenarios iOSCoreCLR Interpreter iOSStripSymbols osx x64 perfiphone12mini net11.0	Log
2890600	dotnet-maui	Controls Layout	Log
1272263	dotnet/maui	Build macOS	Log	dotnet/maui#33198
1272261	dotnet/maui	Controls CollectionView	Log	dotnet/maui#33737
2890682	dotnet-performance	Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main	Log
1272200	dotnet/maui	RunOniOS_MauiNativeAOT macOS	Log
1272201	dotnet/maui	Controls CollectionView	Log
1272185	dotnet/maui	Controls CollectionView	Log
2890599	dotnet-maui	RunOniOS_MauiNativeAOT macOS	Log
2890463	dotnet-maui	Controls Layout	Log
1272086	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT	Log
1272083	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT	Log
1272019	dotnet/maui	Build macOS	Log	dotnet/maui#33774
2890514	dotnet-runtime	Performance ios_scenarios iOSCoreCLR Interpreter iOSStripSymbols osx x64 perfiphone12mini net11.0	Log
1271945	dotnet/maui	Build macOS	Log	dotnet/maui#33765
2890475	dotnet-performance	Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main	Log
1271940	dotnet/performance	Performance blazor_scenarios windows 24H2 x64 Open 9.0	Log
2890462	dotnet-maui	RunOniOS_MauiRelease macOS	Log
1271821	dotnet/runtime	Libraries Test Run checked coreclr osx arm64 Debug	Log	dotnet/runtime#123546
1271823	dotnet/maui	Build macOS	Log	dotnet/maui#33689
1271766	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log
1271610	dotnet/maui	Controls CollectionView	Log
1271735	dotnet/runtime	osx-x64 Release NativeAOT	Log
1271729	dotnet/runtime	iossimulator-x64 Release AllSubsets_Mono_RuntimeTests	Log
2890315	dotnet-runtime	Performance ios_scenarios iOSCoreCLR R2R iOSStripSymbols osx x64 perfiphone12mini net11.0	Log
1271710	dotnet/maui	RunOniOS_MauiRelease macOS-arm64	Log	dotnet/maui#33763
1271699	dotnet/maui	Build macOS	Log	dotnet/maui#33747
1271521	dotnet/runtime	Libraries Test Run release coreclr osx x64 Debug	Log	dotnet/runtime#123471
1271298	dotnet/maui	Controls CollectionView	Log
1271202	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
1271580	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#123680
1271499	dotnet/maui	Build macOS	Log	dotnet/maui#33757
1271524	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
1271425	dotnet/maui	Build macOS	Log	dotnet/maui#33567
1271392	maestro-auth-test/helix-queue-insights-test	HelixTests	Log	maestro-auth-test/helix-queue-insights-test#1115
1270990	dotnet/maui	Controls CollectionView	Log
1271335	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#123695
1271087	dotnet/maui	RunOniOS_BlazorRelease macOS-arm64		dotnet/maui#33747
1271254	dotnet/maui	Build macOS	Log	dotnet/maui#33737
2890080	dotnet-runtime	Performance ios_scenarios iOSCoreCLR R2R iOSStripSymbols osx x64 perfiphone12mini net11.0	Log
1271213	dotnet/maui	RunOniOS_BlazorDebug macOS	Log
1271188	dotnet/maui	Build macOS	Log
1271185	dotnet/maui	Samples macOS	Log	dotnet/maui#33752
2890046	dotnet-performance	Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main	Log
1271123	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1271108	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
2889917	dotnet-maui	Build macOS	Log
1271050	dotnet/maui	RunOniOS_MauiNativeAOT macOS	Log
2889852	dotnet-maui	RunOniOS_BlazorRelease macOS	Log
1270626	dotnet/maui	RunOniOS_MauiNativeAOT macOS-arm64		dotnet/maui#33702
1270759	dotnet/maui	Controls Layout	Log
2889781	dotnet-runtime	Performance ios_scenarios iOSCoreCLR R2R iOSStripSymbols osx x64 perfiphone12mini net11.0	Log
1270758	dotnet/maui	RunOniOS_MauiReleaseTrimFull macOS	Log
2889737	dotnet-performance	Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main	Log
2889696	dotnet-maui	RunOniOS_MauiRelease macOS	Log
1270650	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#123489
1270642	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log	dotnet/runtime#122373
1270637	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#123142
1270632	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#123717
1270617	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#52044
1270471	dotnet/runtime	osx-x64 Debug Installer_Build_And_Test	Log	dotnet/runtime#123645
1270570	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#123694
1270561	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#52664
2889641	dotnet-sdk	Darwin	Log	#57254
2889640	dotnet-sdk	Darwin	Log	#57253
1270547	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#52554
1270534	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#123715
1270531	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit	Log	dotnet/runtime#121956
1270527	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit	Log	dotnet/runtime#123714
1270498	dotnet/maui	Build macOS	Log	dotnet/maui#33719
1270512	dotnet/maui	Controls Layout	Log
1270501	dotnet/performance	Performance imagesharp windows 24H2 x64 Open 9.0	Log	dotnet/performance#5101
1270322	dotnet/runtime	Libraries Test Run checked coreclr osx arm64 Debug	Log	dotnet/runtime#122093
1270185	dotnet/runtime	maccatalyst-x64 Release AllSubsets_Mono	Log
1270422	dotnet/maui	Controls Layout	Log
1270405	dotnet/maui	Controls Layout	Log	dotnet/maui#33737
2889540	dotnet-runtime	Performance ios_scenarios iOSCoreCLR Interpreter iOSStripSymbols osx x64 perfiphone12mini net11.0	Log
1270002	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT_Smoke	Log	dotnet/runtime#123386
1270386	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT_Smoke	Log	dotnet/runtime#123690
1270375	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT_Smoke	Log	dotnet/runtime#123663
2889503	dotnet-performance	Performance maui_scenarios_ios Interpreter CoreCLR osx 15 x64 iPhoneMini12 main	Log
1270337	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#123500
1270319	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#123429
1270289	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT_Smoke	Log	dotnet/runtime#123711
1270236	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#123666
1270232	dotnet/runtime	ios-arm64 Release AllSubsets_Mono_Smoke	Log	dotnet/runtime#123656
Displaying 100 of 852 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
45	349	852

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.