[CI] assembly-processing times out at 600 seconds after all NUnit tests pass

[dalexsoto]

Summary

The T: assembly-processing job intermittently hits the harness's 600-second execution deadline even though the NUnit run completes successfully with all executed tests passing.

• Stable signature: assembly-processing process reaches 600-second harness timeout although NUnit and dotnet exit successfully with all tests passed
• Classification: genuinely flaky test-duration / timeout-budget failure
• Distinct monitored PRs: 4
• Confirmed occurrences: 5
• First observed timeout: 2026-08-31T21:24:56Z
• Latest observed timeout: 2026-09-02T07:20:25Z

The normalized harness result is:

Assembly processing tests: TimedOut (Execution timed out after 10 minutes.)
Executed 235 tasks
Failed: 1
TimedOut: 1
Crashed: 0

Confirmed occurrences

#26020

• Head: 52aa6b725350ec79b8019d3d5e8523c817be9b27
• Build 15180927, attempt 1, assembly-processing / Run tests
• Worker: AcesShared 209
• Task interval: 2026-09-02T07:09:34Z–07:20:25Z
• NUnit XML: 81 total, 80 passed, 1 skipped, 0 failed; result Passed; duration 595.939 seconds.

#26477

• Head: da73a6ac9dad9d6f191770b1eb8316968ecb3af6
• Build 15180868, attempt 1, assembly-processing / Run tests
• Worker: AcesShared 154
• Task interval: 2026-09-02T07:06:11Z–07:18:23Z
• NUnit XML: 81 total, 80 passed, 1 skipped, 0 failed; result Passed; duration 597.213 seconds.

#26495

• Head: 0a76122a3e88077de5c52fc48987562aff5e1aff
• Build 15180799, attempt 1, assembly-processing / Run tests
• Worker: AcesShared 64
• Task interval: 2026-09-02T06:59:20Z–07:11:33Z
• NUnit XML: 81 total, 80 passed, 1 skipped, 0 failed; result Passed; duration 597.279 seconds.

Earlier #26424 occurrences

• Head 1b2392a4fa88d248fdd922275084964afcf6f1d5: build 15157876, attempt 1 timed out on AcesShared 10; the head was subsequently superseded.
• Head b4abc54c0c077b0e1515c5b386a94b723f76f8d1: build 15164848, attempt 1 reproduced the timeout. An externally initiated failed-jobs attempt 2 reran assembly-processing on AcesShared 12 and succeeded at 2026-09-01T10:25:39Z; the full build passed and [net11.0] Merge main into net11.0. #26424 merged.

Passing controls

The same assembly-processing job passed in contemporary monitored builds on multiple workers, including #26021 (AcesShared 79), #26468 (AcesShared 10), #26260 (AcesShared 12), #26464 (AcesShared 179), #26466 (AcesShared 136), #26476 (AcesShared 108), #26526 (AcesShared 46), #26522 (AcesShared 23), and #26498 (AcesShared 10).

The failing PRs are unrelated source changes, and the failure both reproduces and clears across different heads and workers. This rules out a deterministic source regression and supports a load-sensitive timeout-budget flake.

Impact

The test assembly reports a passing NUnit result, but the outer harness marks the job failed because execution and teardown consume the entire 10-minute allowance. This blocks the simulator pipeline and can trigger unnecessary reruns despite no test assertion failure.

Investigation performed

• Inspected Azure build metadata, timelines, job/task logs, and worker assignments.
• Downloaded the per-job TestSummary and nested HtmlReport artifacts for the three current occurrences.
• Parsed the retained NUnit XML and confirmed zero failed tests in every current occurrence.
• Compared the same job across passing monitored builds and the successful [net11.0] Merge main into net11.0. #26424 retry.
• Searched all open and closed dotnet/macios issues separately using the exact timeout text, job name, harness duration, task count, and AssemblyPreparerTests component. No matching issue existed.

The available binlogs cover successful build operations inside the suite; no build error explains the outer harness timeout.

Retry status and workaround

The current #26020, #26477, and #26495 simulator runs are still active, so no retry is permitted until each run is terminal, the head and mergeability are refreshed, retry budgets are reconciled, and no equivalent retry is queued or running.

A failed-jobs rerun can recover, as demonstrated by #26424. The durable fix is to either reduce/parallelize the suite's runtime or give execution and teardown enough headroom beyond the current 600-second deadline.

Azure logs and artifacts are retention-limited.

[dalexsoto]

Active retry status for the assembly-processing 600-second timeout signature:

• PR [dotnet] Detect when the RuntimeIdentifier is changed too late in the build #26477 @ da73a6ac9dad9d6f191770b1eb8316968ecb3af6: Azure accepted build 15180868, simulator_tests attempt 2, rerunning only assembly-processing on AcesShared 4. Retry usage: 1 lifetime / 1 current head.
• PR [bgen] Fix invalid code generation for delegate parameters without [BlockCallback]/[CCallback]. Fixes #10015. #26495 @ 0a76122a3e88077de5c52fc48987562aff5e1aff: Azure accepted build 15180799, simulator_tests attempt 2, rerunning only assembly-processing on AcesShared 11. A separate required request reruns the entire Windows Integration stage for an unrelated failure. Retry usage: 3 lifetime / 2 current head.
• PR [msbuild] Fix multiple resource prefixes support. Fixes #20968 #26020 remains on its original nonterminal run, so no retry has been submitted there.

No equivalent retry will be submitted while attempts are active. The canonical count remains 5 occurrences across 4 PRs pending terminal outcomes. The issue body has no Additional info to collect checklist.

[dalexsoto]

Terminal retry outcomes for the assembly-processing 600-second timeout signature:

• PR [dotnet] Detect when the RuntimeIdentifier is changed too late in the build #26477 @ da73a6ac9dad9d6f191770b1eb8316968ecb3af6: build 15180868, simulator_tests attempt 2, T: assembly-processing succeeded on AcesShared 4 at 2026-09-02T09:44:41Z. The full build then succeeded. Retry usage remains 1 lifetime / 1 current head.
• PR [bgen] Fix invalid code generation for delegate parameters without [BlockCallback]/[CCallback]. Fixes #10015. #26495 @ 0a76122a3e88077de5c52fc48987562aff5e1aff: build 15180799, simulator_tests attempt 2, T: assembly-processing succeeded on AcesShared 11 at 2026-09-02T09:45:06Z. Its separately requested full Windows Integration retry remains active. Retry usage remains 3 lifetime / 2 current head.

The timeout did not recur in either targeted rerun, supporting the genuinely flaky outer-harness/teardown classification. No source change was needed. Canonical count remains 5 confirmed occurrences across 4 PRs, and the issue body has no Additional info to collect checklist.

[dalexsoto]

New timeout-budget variant and retry outcome:

• PR [main] Update dependencies from dotnet/macios #26519 @ 4458562853d7b5c27e2dd30fdb8bd7b4e404e125: build 15182211, attempt 1, T: assembly-processing / Run tests hit the same fixed 600-second execution deadline on AcesShared 17.
  • This materially extends the signature: the runner was killed before producing NUnit XML after 71 passed, 1 skipped, and 0 failed cases (72 of the expected 81 completed), rather than just after a passing NUnit result.
  • The last completed events were BindingResourcePackageRoundTrip(iOS,False,True) passing and FromBinlog skipping; there was no assertion failure or crash report.
  • The same PR's preceding package-update head passed this job in 478 seconds on AcesShared 63. Contemporary controls passed in 466–502 seconds on AcesShared 117, AcesShared 8, and AcesShared 211. This remains the same load-sensitive timeout-budget root, preserved as a distinct stable variant in the occurrence ledger.
  • Build 15182211 remains active, so no retry is permitted yet.
• PR [msbuild] Fix multiple resource prefixes support. Fixes #20968 #26020 @ 52aa6b725350ec79b8019d3d5e8523c817be9b27: build 15180927, simulator_tests attempt 2, T: assembly-processing succeeded on AcesShared 4 at 2026-09-02T10:12:37Z; the simulator stage succeeded at 10:12:44Z. Its separately requested Windows and API Scan retries remain active. Retry usage remains 3 lifetime / 3 current head.

The reconciled canonical total is now 6 confirmed occurrences across 5 distinct PRs. The issue body has no Additional info to collect checklist.

[dalexsoto]

Fix PR status:

• PR #26532 @ 07900653488af66a5a9651595d4925c2fa5679b5 changes tests/xharness/Harness.cs to raise the assembly-processing timeout from 10 minutes to 30 minutes.
• The PR explicitly fixes [CI] assembly-processing times out at 600 seconds after all NUnit tests pass #26528 and is currently mergeable with CI active: 20 successful, 3 skipped, 22 running, and 2 queued checks at the latest snapshot.
• No source changes were made by the monitor; this is a read-only status record for Rolf's branch.

The fix is not yet considered validated until its assembly-processing job and full CI finish successfully. The issue body has no Additional info to collect checklist.

[dalexsoto]

Fix validation and active retry status:

• PR #26532 @ 07900653488af66a5a9651595d4925c2fa5679b5: build 15182795, T: assembly-processing passed on AcesShared 90 after 21m44s (11:08:12Z–11:29:56Z). This would have exceeded the old 10-minute limit and directly validates the new 30-minute timeout for this job. Full PR CI remains active.
• PR [main] Update dependencies from dotnet/macios #26519 @ 4458562853d7b5c27e2dd30fdb8bd7b4e404e125: build 15182211 became terminal with only the known assembly-processing timeout. After reconciling one historical external retry on an older head, Azure accepted targeted simulator_tests attempt 2; only assembly-processing is pending. Usage is 2 PR-lifetime / 1 current-head request.

Canonical failure count remains 6 occurrences across 5 PRs pending the retry and full fix-PR outcomes. The issue body has no Additional info to collect checklist.

[dalexsoto]

Terminal retry outcome for PR #26519 @ 4458562853d7b5c27e2dd30fdb8bd7b4e404e125:

• Build 15182211, simulator_tests attempt 2, T: assembly-processing succeeded on AcesShared 49 at 2026-09-02T12:06:20Z; the simulator stage succeeded at 12:06:25Z.
• The fixed 600-second timeout did not recur. Result publication is still completing.
• Retry usage remains 2 PR-lifetime / 1 current-head request, including the reconciled historical retry on an older head.

Canonical count remains 6 confirmed occurrences across 5 PRs. PR #26532's 30-minute timeout fix also passed its assembly-processing job after 21m44s; its full CI remains active. The issue body has no Additional info to collect checklist.

[dalexsoto]

New confirmed occurrence on PR #26535 @ 3d2e0c0665e69b90d74f2b5d9a186d9b475a283f:

• Build 15183044, attempt 1, T: assembly-processing / Run tests hit the fixed 600-second timeout on AcesShared 65.
• Retained NUnit XML reports 81 total, 80 passed, 1 skipped, 0 failed, result Passed, duration 595.161 seconds. The outer harness then marked Assembly processing tests: TimedOut.
• The PR changes only AppKit XML documentation and the documentation known-failures baseline, so this is unrelated to its source changes and exactly matches the original timeout-budget signature.
• Build 15183044 remains active, so no retry is permitted yet.

The reconciled canonical total is now 7 confirmed occurrences across 6 distinct PRs. PR #26532's 30-minute timeout fix has already passed this job after 21m44s. The issue body has no Additional info to collect checklist.

[dalexsoto]

Final fix outcome:

• PR #26532 @ 07900653488af66a5a9651595d4925c2fa5679b5 merged at 2026-09-02T13:37:22Z, automatically closing this issue as completed.
• Build 15182795, T: assembly-processing passed on AcesShared 90 after 21m44s, directly validating the new 30-minute timeout.
• The overall build was partiallySucceeded only because the unrelated Sonoma Clean bot timeout tracked by [CI] Sonoma Clean bot hangs in simctl delete all for 60 minutes #26525 was nonblocking. No assembly-processing failure remained.

The canonical history remains 7 confirmed occurrences across 6 PRs. The completed close reason remains valid, so the issue is not reopened. The issue body has no Additional info to collect checklist.

[dalexsoto]

Active retry status for the pre-fix occurrence on PR #26535 @ 3d2e0c0665e69b90d74f2b5d9a186d9b475a283f:

• After the full run became terminal, Azure accepted build 15183044 simulator_tests attempt 2, rerunning only T: assembly-processing; the job is pending.
• A separate request reruns the unrelated Sonoma archive-corruption failure tracked by [CI] Sonoma corrupts mac-test-package.7z during test expansion #26536.
• Retry usage is 2 PR-lifetime / 2 current-head requests across both accepted stage requests.

This occurrence ran before #26532's 30-minute timeout fix merged, so it does not invalidate this issue's completed close reason and the issue remains closed. The issue body has no Additional info to collect checklist.