[CI] AcesShared agent hangs until job timeout and ignores cancellation

[dalexsoto]

Failure signature

An AcesShared agent becomes unresponsive during an active task:

• The task produces no finalized Azure DevOps log.
• The job reaches the 180-minute hard limit.
• Azure reports that the agent did not respond to cancellation within five minutes.
• Post-job diagnostics and artifact publication cannot complete reliably.

Stable signature:

AcesShared agent becomes unresponsive during an active task, produces no finalized task log, exceeds the 180-minute job limit, and ignores the 5-minute cancellation request

Confirmed occurrences

PR #26424

• Head: b4abc54c0c077b0e1515c5b386a94b723f76f8d1
• Build: 15164848, attempt 1, T: framework / Install dotnet workloads
• The attempt-1 task log was not finalized and is no longer retained.
• Azure reported the 180-minute timeout and unresponsive cancellation behavior.
• Failed-jobs attempt 2 reran T: framework on AcesShared 38 and succeeded from 2026-09-01T10:09:18Z to 10:20:06Z; build 15164848 completed successfully at 10:29:58Z.
• Worker identity for the failed attempt is unavailable from retained authorized diagnostics.

PR #26568

• Head: 4237a344488a1ab2c7cee633ec9b95fa8e9f7afb
• Build: 15218580, attempt 1, Build macOS tests / Build
• Worker: AcesShared 4
• Job started at 2026-09-04T20:10:11Z; Build started at 20:17:12Z.
• The Build task produced no finalized log. Azure failed the stage at 23:15:12Z after the 180-minute cap and reported that the agent ignored cancellation for five minutes.
• Package macOS tests never ran, so the downstream macOS 27 test stage could not start.
• Retry is deferred until build 15218580 becomes terminal. Current usage is 0/25 PR lifetime and 0/10 for the current head.

First observed: 2026-09-01T07:00Z (attempt-1 failure window; exact task completion unavailable because the agent stopped reporting).

Latest observed: 2026-09-04T23:15:12Z.

Recurrence: 2 jobs across 2 distinct monitored PRs.

Classification and impact

This is a CI infrastructure/agent failure, not a source regression:

• It occurred in unrelated tasks (Install dotnet workloads and Build) on unrelated merge PRs.
• Both tasks stopped reporting without a finalized task log.
• Azure emitted the same job-limit and cancellation-timeout symptoms.
• The current PR changes only Xtro baseline files, and the same macOS build job previously completed successfully in about 36 minutes on build 15214043.

The failure blocks downstream stages and can suppress the diagnostic artifacts needed to identify a task-level cause.

Investigation performed

• Inspected Azure timelines, task records, job issues, worker assignments, and available task logs.
• Compared the two stable signatures and rejected task-specific issues:
  • [CI] Provisionator Xcode downloads at ~160 KB/s hit the 180-minute job limit #26212 tracks visibly slow Xcode downloads.
  • [CI] Provisionator XcodeSelect subprocess hangs until the 180-minute job cap #26226 tracks identified Provisionator XcodeSelect subprocess hangs.
  • [CI] xcodebuild -downloadPlatform tvOS hangs until test-job timeout #26533 tracks an identified xcodebuild -downloadPlatform tvOS hang.
• Searched all open and closed dotnet/macios issues using the exact cancellation message, job names, 180-minute symptom, and missing-task-log evidence.

Workaround

Rerun failed jobs after the build is terminal and normal head, mergeability, active-attempt, and retry-budget gates permit it. A durable fix needs agent-side diagnostics or watchdog handling that can survive an unresponsive task process and preserve the last task state.

Azure logs and artifacts are retention-limited.

[dalexsoto]

Retry update for PR #26568 at head 4237a344488a1ab2c7cee633ec9b95fa8e9f7afb:

• Build 15218580 became terminal failed at 2026-09-04T23:19:49Z.
• After refreshing the unchanged conflict-free head, terminal timeline, zero prior retries, and equivalent-attempt state, Azure accepted build_macos_tests attempt 2 with failed-jobs scope.
• Build macOS tests attempt 2 is pending.
• The recurring count remains 2 confirmed jobs across 2 monitored PRs pending the retry outcome.
• A separate accepted request reruns the unrelated monotouch_ios NSUrlSession stall tracked by [CI] iOS 27 monotouch-test stalls in NSUrlSession redirect tests with simulator daemon crash loops #26221. Reconciled usage is 2/25 PR lifetime and 2/10 for this head.
• Issue-specific checklist: none; the issue body has no Additional info to collect section.

[dalexsoto]

Retry outcome for PR #26568 at head 4237a344488a1ab2c7cee633ec9b95fa8e9f7afb:

• Build 15218580, build_macos_tests attempt 2 succeeded on AcesShared 2.
• The job ran from 2026-09-04T23:22:34Z to 23:51:44Z; the stage completed successfully at 23:51:58Z.
• The unresponsive-agent signature did not recur. All four downstream macOS test stages started successfully.
• Retry usage remains 2/25 PR lifetime and 2/10 for this head.
• The recurring count remains 2 confirmed jobs across 2 monitored PRs.
• Issue-specific checklist: none; the issue body has no Additional info to collect section.

[dalexsoto]

New confirmed occurrence on PR #26568 at head b06e22cade15faec3ab8b86dd82d8c911ea9a55a:

• Build 15222573, attempt 1, T: dotnettests_maccatalyst / Run tests
• Worker: AcesShared 35. The job ran from 2026-09-05T01:05:19Z until 04:13:58Z; Run tests started at 01:17:43Z.
• Azure terminated the job after the 180-minute cap and reported that the agent did not respond to cancellation within five minutes. The active task had no finalized task log; TestSummary, agent-log, and post-job artifacts were abandoned or unavailable.
• Classification: exact recurring unresponsive-agent infrastructure signature. Reconciled count: 3 confirmed jobs across 2 monitored PRs.
• After refreshing the unchanged conflict-free head and terminal build, Azure accepted simulator_tests attempt 2 with failed-jobs scope. It is pending for T: dotnettests_maccatalyst and the unrelated T: monotouch_ios flake tracked by [CI] iOS 27 monotouch-test stalls in NSUrlSession redirect tests with simulator daemon crash loops #26221.
• Retry usage is now 3/25 PR lifetime and 1/10 for this head.
• Issue-specific checklist: none; the issue body has no Additional info to collect section.

[dalexsoto]

Retry outcome for PR #26568 at head b06e22cade15faec3ab8b86dd82d8c911ea9a55a:

• Build 15222573, T: dotnettests_maccatalyst attempt 2 succeeded on AcesShared 1 at 2026-09-05T06:29:02Z.
• The unresponsive-agent signature did not recur. The canonical count remains 3 confirmed jobs across 2 monitored PRs.
• The same simulator_tests retry request is still running the unrelated monotouch_ios NSUrlSession flake tracked by [CI] iOS 27 monotouch-test stalls in NSUrlSession redirect tests with simulator daemon crash loops #26221.
• Retry usage remains 3/25 PR lifetime and 1/10 for this head.
• Issue-specific checklist: none; the issue body has no Additional info to collect section.