slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2706613	dotnet-install-scripts	MacOSTests	Log
2706498	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
2706392	dotnet-performance	Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main	Log
1038542	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
2706262	dotnet-runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
2706278	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2706123	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
2706070	dotnet-performance	Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main	Log
2705981	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2705809	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
2705687	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
2705649	dotnet-performance	Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main	Log
2705496	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2705394	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
2705215	dotnet-sdk	AoT: macOS (x64)	Log	#49771
1037417	dotnet/runtime	osx-x64 Debug PALTests	Log
2705261	dotnet-performance	Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main	Log
1037222	dotnet/runtime	Libraries Test Run release coreclr osx x64 Debug	Log	dotnet/runtime#115395
1037213	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#115345
1037253	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#48815
1037201	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log
1037194	dotnet/runtime	Libraries Test Run release coreclr osx x64 Release	Log
2705177	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
1037196	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log
1037237	dotnet/runtime	maccatalyst-x64 Release AllSubsets_NativeAOT_RuntimeTests_AppSandbox	Log
2705142	dotnet-performance	Performance maui_scenarios_ios FullAOT Mono osx 15 x64 iPhoneMini12 main	Log
1037184	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#115310
1037199	dotnet/runtime	Libraries Test Run release coreclr osx x64 Release	Log
1037217	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#48877
1037148	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#115408
1037138	dotnet/runtime	Libraries Test Run release coreclr osx x64 Release	Log
1037154	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#115410
2705076	dotnet-sdk	AoT: macOS (x64)	Log	#49771
1036923	dotnet/runtime	osx-x64 Release NativeAOT	Log	dotnet/runtime#115388
2704974	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2704824	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
2704795	dotnet-performance	Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main	Log
2704664	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2704417	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
1036464	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#115388
2704277	dotnet-performance	Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main	Log
2704195	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
1036144	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
2704105	dotnet-performance	Performance maui_scenarios_ios FullAOT Mono osx 15 x64 iPhoneMini12 main	Log
2703973	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2703939	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
2703877	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
1035714	dotnet/performance	Performance powershell ubuntu 2204 x64 Open main	Log
2703856	dotnet-performance	Performance maui_scenarios_ios FullAOT Mono osx 15 x64 iPhoneMini12 main	Log
2703828	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2703695	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
2703546	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2703441	dotnet-performance	Performance maui_scenarios_ios FullAOT Mono osx 15 x64 iPhoneMini12 main	Log
2703366	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
1034912	maestro-auth-test/helix-queue-insights-test	HelixTests	Log	maestro-auth-test/helix-queue-insights-test#996
2703188	dotnet-performance	Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main	Log
2703108	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
2702922	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
2702926	dotnet-performance	Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main	Log
2702692	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
1034380	maestro-auth-test/helix-queue-insights-test	HelixTests	Log	maestro-auth-test/helix-queue-insights-test#995
2702552	dotnet-performance	Performance nativeaot_scenarios windows 20H1 arm64 Tiger 8.0	Log
2702532	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
1034085	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
2702413	dotnet-performance	Performance maui_scenarios_ios FullAOT Mono osx 15 x64 iPhoneMini12 main	Log
2702356	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2702097	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
2701979	dotnet-performance	Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main	Log
2701346	dotnet-performance	Performance maui_scenarios_ios FullAOT Mono osx 15 x64 iPhoneMini12 main	Log
2701155	dotnet-performance	Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main	Log
2701106	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
2701009	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
2700873	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
2700836	dotnet-performance	Performance maui_scenarios_ios FullAOT Mono osx 15 x64 iPhoneMini12 main	Log
2700704	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2700559	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2700461	dotnet-performance	Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main	Log
2700151	dotnet-performance	Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main	Log
2700075	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2700037	dotnet-performance	Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main	Log
2699915	dotnet-runtime	Performance ios_scenarios iOSNativeAOT JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2699820	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2699796	dotnet-performance	Performance maui_scenarios_ios FullAOT Mono osx 15 x64 iPhoneMini12 main	Log
1031449	dotnet/performance	Performance micro windows 22H2 x86 Open 9.0	Log
2698950	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
2698745	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
2698694	dotnet-performance	Performance maui_scenarios_ios FullAOT Mono osx 15 x64 iPhoneMini12 main	Log
2698523	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
2698356	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2698226	dotnet-performance	Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main	Log
2698215	dotnet-runtime	Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0	Log
1029500	dotnet/runtime	osx-arm64 Debug PALTests	Log
2698038	dotnet-runtime	Performance ios_scenarios iOSNativeAOT JIT osx x64 perfiphone12mini net10.0	Log
2697888	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2697822	dotnet-performance	Performance maui_scenarios_ios FullAOT Mono osx 15 x64 iPhoneMini12 main	Log
2697591	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2697332	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
2697295	dotnet-performance	Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main	Log
2696936	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0	Log
2696918	dotnet-performance	Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main	Log
Displaying 100 of 214 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	61	214

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.