[CI] Sonoma Clean bot hangs in simctl delete all for 60 minutes

[dalexsoto]

Summary

The Sonoma Clean bot task has hung in simctl delete all until Azure DevOps terminates the task at its 60-minute timeout in two distinct monitored PRs.

• Stable signature: X64 Mac Sonoma clean-bot.sh hangs in simctl delete all until the 60-minute Azure task timeout
• Classification: CI infrastructure failure
• Distinct monitored PRs: 2
• Confirmed occurrences: 2
• First observed timeout: 2026-09-01T18:48:27Z
• Latest observed timeout: 2026-09-02T03:55:54Z
• Worker in both occurrences: XAMBOT-1024.Sonoma

Confirmed occurrences

#26475

• Head: 2ddfa5661d7ae0592ef92481927aa61f28ba8df5
• Pipeline: xamarin-macios-sim-pr-tests
• Build 15169876, attempt 1, X64 Mac Sonoma / Clean bot
• Task interval: 2026-09-01T17:48:17Z–18:48:27Z
• Minimal signature:
  • + /Applications/Xcode.app/Contents/Developer/usr/bin/simctl delete all
  • The task has timed out.
• Azure build result: partiallySucceeded.
• Retry outcome: no cleanup retry was submitted. A separate permitted retry addressed the unrelated FinalizationRaceCondition test failure and passed; the cleanup timeout remained as a nonblocking warning.

#26484

• Head: fd099f147cffffbce1c6a10b5861f4172fd8117b
• Pipeline: xamarin-macios-sim-pr-tests
• Build 15178389, attempt 1, X64 Mac Sonoma / Clean bot
• Task interval: 2026-09-02T02:55:44Z–03:55:54Z
• Minimal signature:
  • + /Applications/Xcode.app/Contents/Developer/usr/bin/simctl delete all
  • The task has timed out.
• Azure build result: partiallySucceeded; all GitHub checks are green.
• Retry outcome: none. The timeout was nonblocking and no failed test or build job remained to rerun.

Reproduction pattern and impact

Both occurrences used the same job, task identifier, command, operating-system lane, and worker. The command produced no completion output for about 60 minutes and occupied the agent until Azure timed it out. Tests were not invalidated, but the task delayed each pipeline and left the Azure build partially successful.

Investigation performed

The Azure build metadata, timelines, task records, and Clean bot logs were compared directly. The exact command, task timeout text, task duration, and worker match across both PRs. Searches across all open and closed dotnet/macios issues used the exact command, task name, timeout text, worker, script name, and simulator-cleanup terms. The only candidates were #6740, an unrelated Xcode UI synchronization report, and #26385, an unrelated checkout/credential failure.

A binlog, TestSummary, HtmlReport, or crash report is not applicable because this is a shell-based post-test simulator cleanup hang rather than a build, test assertion, or process crash.

Current diagnosis and workaround

This is a worker-specific CoreSimulator cleanup infrastructure failure. No source workaround is known. The affected worker's CoreSimulator state should be inspected or reset, and the cleanup path should capture bounded diagnostics before timing out so future hangs identify the blocking simulator service or process.

Azure logs are retention-limited.

[dalexsoto]

New distinct-PR occurrence for the Sonoma Clean bot / simctl delete all timeout:

• PR [bgen] Fix invalid code generation for delegate parameters without [BlockCallback]/[CCallback]. Fixes #10015. #26495 @ 0a76122a3e88077de5c52fc48987562aff5e1aff
• Build 15180799, attempt 1, X64 Mac Sonoma / Clean bot
• Worker: XAMBOT-1028.Sonoma
• Task interval: 2026-09-02T07:29:29Z–08:29:35Z
• Exact command: /Applications/Xcode.app/Contents/Developer/usr/bin/simctl delete all
• Terminal symptom: The task has timed out. after 60 minutes
• Classification: CI infrastructure failure

The cleanup failure is nonblocking (succeededWithIssues), but the same simulator build also has the separately tracked #26528 assembly-processing timeout and a Windows remote-test failure. Build 15180799 remains active, so no retry has been submitted.

The canonical total is now 3 confirmed occurrences across 3 distinct monitored PRs. The issue body has no Additional info to collect checklist.

[dalexsoto]

New confirmed occurrence on PR #26532 @ 07900653488af66a5a9651595d4925c2fa5679b5:

• Build 15182795, Sonoma / Clean bot ran on XAMBOT-1007.Sonoma.
• The task reached /Applications/Xcode.app/Contents/Developer/usr/bin/simctl delete all at 2026-09-02T11:36:01Z and produced no further progress until the 60-minute task timeout at 12:36:11Z.
• Azure marked the cleanup task/stage succeededWithIssues, so the build completed partiallySucceeded. The assembly-processing timeout fix itself passed, and PR [xharness] Bump assembly-preparer test timeout to 30 minutes. #26532 merged at 13:37:22Z; no cleanup retry is warranted.

The canonical total is now 4 confirmed occurrences across 4 distinct PRs. The issue body has no Additional info to collect checklist.

[dalexsoto]

New distinct-PR occurrence on PR #26513 @ a354efde93df4b8e7ad9eb15a88ebd64916b2be1:

• Build 15184608, attempt 1, X64 Mac Sonoma / Clean bot ran on XAMBOT-1028.Sonoma.
• The task reached /Applications/Xcode.app/Contents/Developer/usr/bin/simctl delete all at 2026-09-02T16:12:38Z and produced no further progress until Azure reported The task has timed out. at 17:12:47Z.
• Classification: CI infrastructure failure. The cleanup task was succeededWithIssues; this Sonoma job separately failed RegistrarTest.TestRetainReturnValue, which passed repeatedly in same-build control lanes.
• Build 15184608 remains active, so no retry has been submitted. Any retry decision is deferred until the run is terminal and the head, mergeability, active attempts, and retry counters are refreshed.

The canonical total is now 5 confirmed occurrences across 5 distinct monitored PRs. The issue body has no Additional info to collect checklist.