[CI] Windows Integration Dotnet tests exceeds the 120-minute job limit while progressing

[dalexsoto]

Summary

The Windows Integration Tests / Dotnet tests job has reached its fixed 120-minute Azure DevOps limit in two distinct monitored PRs while test work was either still progressing or had completed successfully. Azure then canceled the remaining remote-test or artifact-publication work.

• Stable signature: Windows Integration Dotnet tests reaches the fixed 120-minute job limit while test workflow remains successful or actively progressing, canceling remaining test or publication work
• Classification: CI infrastructure / job-duration budget failure
• Distinct monitored PRs: 2
• Confirmed occurrences: 2
• First confirmed occurrence: 2026-09-02T12:05:36Z
• Latest confirmed occurrence: 2026-09-02T17:46:07Z

Confirmed occurrences

PR #26020

• Head: 52aa6b725350ec79b8019d3d5e8523c817be9b27
• Pipeline: xamarin-macios-sim-pr-tests
• Build 15180927, Windows Integration attempt 2
• Worker: VSM-XAM-136
• Job interval: 2026-09-02T10:05:37Z–12:05:36Z
• Download artifacts took 1,004.7 seconds, local tests took 619.0 seconds, and remote tests took 4,422.7 seconds. Those tasks succeeded, but the job limit canceled Publish to Artifact Services Drop.
• A full Windows Integration attempt 3 passed on VSM-XAM-124; build 15180927 succeeded and the PR later merged.

PR #26513

• Head: a354efde93df4b8e7ad9eb15a88ebd64916b2be1
• Pipeline: xamarin-macios-sim-pr-tests
• Build 15184608, attempt 1, Windows Integration / Dotnet tests
• Worker: VSM-XAM-129
• Job interval: 2026-09-02T15:45:52Z–17:46:07Z
• Run .NET tests remotely ran from 16:37:57Z until cancellation. Its log contains 31 completed passing test records and no test failures. The last recorded test passed at 17:44:20Z, only 86 seconds before Azure began cancellation at 17:45:46Z.
• Build 15184608 is still active, so no retry has been submitted. Repository policy requires rerunning the entire Windows Integration stage when eligible.

Impact and reproduction pattern

The test workload duration varies enough that the 120-minute job limit can terminate otherwise healthy CI. Depending on timing, either tests remain incomplete or post-test result and artifact publication is skipped. This consumes the Windows and paired Mac agents for two hours and requires a full-stage retry.

Investigation

The Azure build metadata, stage and job timelines, task durations, and remote-test logs were compared directly. Neither occurrence contains a deterministic test failure responsible for cancellation. The #26513 remote-test log continued reporting passes immediately before the deadline; #26020 had already completed both local and remote tests.

Separate searches across all open and closed issues used the exact Azure timeout message, stage and job names, Run .NET tests remotely, worker names, publication task, and Windows timeout terms. The only candidate was #26457, which is an unrelated Ventura Show Bot Info / ioreg pre-test stall.

Current diagnosis and workaround

This is a CI duration-budget failure rather than a source regression. The current workaround is to rerun the entire Windows Integration stage after the parent build is terminal and retry policy permits it. Longer-term options are to increase the job budget, reduce setup and artifact-download time, or split the remote test workload so normal duration variance cannot consume the full job allowance.

Azure logs and artifacts are retention-limited.

[dalexsoto]

Retry update for PR #26513 @ a354efde93df4b8e7ad9eb15a88ebd64916b2be1:

• Build 15184608 completed attempt 1 failed, with the Windows Integration Dotnet tests job canceled at its 120-minute limit while remote tests were still progressing.
• After refreshing the unchanged head, conflict-free mergeability, terminal build, all four observable automatic PR builds, stage attempt history, and retry ledger, the monitor requested the required entire Windows Integration stage retry using Azure DevOps (forceRetryAllJobs: true).
• Azure accepted the request and created Windows Integration attempt 2. The stage is currently active.
• Observable accepted retry usage for this PR/head is 1 lifetime / 1 for this head. No prior manual or programmatic retries were found across the four PR builds.

The issue body has no Additional info to collect checklist. The outcome will be reported when attempt 2 reaches a terminal state.

[dalexsoto]

Retry outcome for PR #26513 @ a354efde93df4b8e7ad9eb15a88ebd64916b2be1:

• Windows Integration attempt 2 succeeded in build 15184608.
• Dotnet tests ran on VSM-XAM-136 from 2026-09-02T18:35:34Z to 20:24:14Z and completed successfully.
• The full Windows Integration stage, including Mac reservation and re-enable phases, completed successfully at 20:26:52Z.
• The original fixed-120-minute failure did not recur. Retry accounting remains 1 accepted lifetime / 1 accepted for this head.
• Build 15184608 completed failed only because unrelated simulator and Sonoma failures remain; the Windows check is green.

The issue body has no Additional info to collect checklist.

[dalexsoto]

New distinct-PR occurrence and active retry for the Windows duration-budget signature:

• PR [dotnet] Remove stale template packs when installing. Fixes #26540. #26551, head 32f4b9bd6961144d7d9def785863bb20dcee674a, build 15195438 attempt 1 / Windows Dotnet tests, on VSM-XAM-126.
• The job ran from 2026-09-03 05:58:58 to 07:59:16 UTC and Azure canceled it at the 120-minute limit. Run .NET tests remotely was still active from 06:37:56 to 07:59:02 UTC.
• The exact extension-build test that failed on [main] Update dependencies from dotnet/macios #26552 passed here, and later RemoteTest work was still passing at 07:55:48 UTC. No deterministic test failure preceded cancellation.
• An externally initiated full Windows Integration attempt 2 started at 12:08:18 UTC and is currently reserving its macOS worker. The retry actor is unavailable from Azure stage history; the monitor did not submit this request.
• Retry usage is 1 lifetime / 1 head.

The canonical total is now 3 confirmed occurrences across 3 distinct monitored PRs. The issue body has no Additional info to collect checklist.

[dalexsoto]

Successful retry outcome for PR #26551, head 32f4b9bd6961144d7d9def785863bb20dcee674a:

• Build 15195438, Windows Integration attempt 2 completed successfully.
• Dotnet tests ran on VSM-XAM-133 from 2026-09-03 12:16:22 to 14:10:30 UTC and succeeded.
• The full Windows Integration stage completed successfully at 14:13:03 UTC, result publication succeeded, and the parent build completed successful at 14:16:36 UTC.
• The fixed-120-minute duration-budget failure did not recur. Retry usage remains 1 lifetime / 1 head.

The canonical count remains 3 confirmed occurrences across 3 distinct monitored PRs. The issue body has no Additional info to collect checklist.

[dalexsoto]

Final merge outcome for PR #26551:

• The PR merged at 2026-09-03 16:16:54 UTC on head 32f4b9bd6961144d7d9def785863bb20dcee674a with 51 successful and 3 skipped checks.
• The successful full Windows Integration attempt 2 remained green through merge, and the fixed-120-minute duration-budget failure did not recur.
• Retry usage remains 1 lifetime / 1 head; the canonical count remains 3 confirmed occurrences across 3 distinct monitored PRs.

The issue remains open for the underlying CI duration-budget risk. The issue body has no Additional info to collect checklist.