Skip to content

ci: make dependency resolution resilient and add an advisory Xcode preview build - #4272

Merged
harsh62 merged 1 commit into
mainfrom
ci/resilient-dependency-resolution
Aug 13, 2026
Merged

ci: make dependency resolution resilient and add an advisory Xcode preview build#4272
harsh62 merged 1 commit into
mainfrom
ci/resilient-dependency-resolution

Conversation

@harsh62

@harsh62 harsh62 commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Issue

Jobs were being killed at the job timeout having produced no build output at all, with cleanup terminating orphaned git and git-remote-http processes. They were not stuck on a test or a build — they were still fetching dependencies.

Left to xcodebuild, resolution happens silently inside the build/test invocation. SwiftPM checks out ~31 repositories one at a time through blocking git subprocesses and prints nothing until the first compile, so a slow clone is indistinguishable from a hang. I confirmed the mechanism by sampling a stalled process:

Workspace.loadPackageGraph → Workspace._resolve
  → Workspace.updateDependenciesCheckouts → Workspace.checkoutRepository
    → GitRepository.checkout(revision:) → AsyncProcess.waitUntilExit()

Description

Resolution is now explicit, bounded, and retried

Added as a step in both run_xcodebuild and run_xcodebuild_test:

  • Bounded per attempt by a background watchdog, so a stalled clone is killed instead of consuming the job budget
  • Retried up to 3 times, clearing partial mirror state between attempts (a half-written clone is a common way for this to wedge)
  • Logged in its own group with elapsed time, so "slow" is distinguishable from "stuck"
  • Fails explicitlythis is a dependency fetch problem, not a test failure — rather than as a bare timeout

The build and test invocations now always pass -disableAutomaticPackageResolution. Whether checkouts came from a warm cache or from the step above, xcodebuild can never fall back to resolving mid-run. That fallback is what produced the silent hangs.

Implementation note: the bound uses a watchdog because timeout-minutes: is rejected on steps inside a composite action, and timeout(1) is not installed on macOS runners.

Timeouts 30 → 60 minutes

Including the eight callers that hardcoded 30 and so overrode the reusable-workflow default. This is headroom, not the fix — for cold fetches and for simulator startup, which I measured taking tens of minutes on visionOS/iOS. Resolution is separately bounded above, so a genuine stall still fails fast.

Advisory Xcode 27 preview build

Builds against the xcode-27 preview image on PRs and pushes to main, so next-Xcode breakages surface on the change that introduces them. Every job sets continue-on-error — a beta toolchain on a capacity-constrained preview image is not a suitable required check — and a summary step says so in the run summary.

It bypasses get_platform_parameters on purpose (that action allowlists only officially supported Xcode versions; a beta is outside that set), uses a generic iOS destination rather than a pinned device name that would break on the next beta rollout, and treats xcbeautify as optional since the preview image is built separately from the standard macOS images.

Deliberately NOT included: cache seeding

An earlier attempt added a workflow to seed the shared SwiftPM cache. I dropped it: a seeding workflow only runs once it is on the default branch, and GitHub scopes branch caches so a cache seeded on one branch is invisible to sibling PRs. That made it unverifiable before merge and useless to the PRs that needed it.

This change instead makes the cold path work on its own. Caching stays a speed optimisation rather than a correctness requirement.

How did you test these changes?

Verified against a completely empty cache — I deleted every amplify-packages-* entry first, so this is a genuine cold start:

Phase Result
xcodebuild -resolvePackageDependencies 2m11s, 31 checkouts
xcodebuild test with resolution disabled 1m43s, ** TEST SUCCEEDED **, 31 tests
Fetch attempts during the test 0

Also checked whether swift package resolve would be a faster substitute — it is not (4m19s vs xcodebuild's 2m11s), so xcodebuild is retained.

The watchdog itself was unit-checked locally across all three paths: fast success returns 0, fast failure returns non-zero, and a hang is killed at the bound rather than running to completion.

All 76 workflow/action YAML files parse.

Documentation

No customer-facing changes — CI configuration only. The rationale for each non-obvious choice (why resolution is a separate step, why the watchdog, why the timeout value, why the cache key is not the fix) is recorded in comments beside the code rather than only here.

Checklist

  • PR description included
  • Unit tests added/updated — not applicable; CI configuration, validated by YAML parsing plus the cold-start measurement above
  • Documentation updated — inline comments

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Jobs were being killed at the job timeout with no build output at all, and
cleanup terminating orphaned git and git-remote-http processes. They were not
hung on a test or a build — they were still fetching dependencies.

Left to xcodebuild, resolution happens silently inside the build or test
invocation. SwiftPM checks out ~31 repositories one at a time through blocking
git subprocesses and prints nothing until the first compile, so a slow clone is
indistinguishable from a hang: the job emits no output, and is eventually killed
with the whole run lost.

Resolution is now an explicit step in both run_xcodebuild and
run_xcodebuild_test:

  - bounded per attempt by a background watchdog, so a stalled clone is killed
    rather than left to consume the job budget
  - retried up to 3 times, clearing partial mirror state between attempts, since
    a half-written clone is a common way for this to wedge
  - logged in its own group with elapsed time, so slow is distinguishable from
    stuck
  - failing with an explicit "this is a dependency fetch problem, not a test
    failure" rather than a bare timeout

The build and test invocations now always pass
-disableAutomaticPackageResolution. Whether the checkouts came from a warm cache
or from the step above, xcodebuild can never fall back to resolving mid-run,
which is the fallback that produced the silent hangs.

The watchdog is used because `timeout-minutes:` is rejected on steps inside a
composite action and `timeout(1)` is not installed on macOS runners.

Job timeouts go from 30 to 60 minutes, including the eight callers that
hardcoded 30 and so overrode the reusable workflow default. That is headroom for
cold fetches and for simulator startup, which has been observed taking tens of
minutes on visionOS and iOS; it is not the fix, since resolution is separately
bounded above.

Deliberately not included: cache seeding. A seeding workflow only runs once it
is on the default branch, and GitHub scopes branch caches so a cache seeded on
one branch is invisible to sibling PRs. That made it unverifiable before merge
and useless to the PRs that needed it. This change instead makes the cold path
work on its own; caching remains a speed optimisation rather than a correctness
requirement.

Verified locally against a completely empty cache (all amplify-packages entries
deleted first): resolve took 2m11s and produced 31 checkouts, then
`xcodebuild test` with resolution disabled took 1m43s and passed
(** TEST SUCCEEDED **, 31 tests), with zero fetch attempts during the test.
Also confirmed `swift package resolve` is not a faster substitute — it took
4m19s against xcodebuild's 2m11s.
@harsh62
harsh62 requested a review from a team as a code owner August 12, 2026 23:05
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@harsh62
harsh62 temporarily deployed to IntegrationTest August 12, 2026 23:05 — with GitHub Actions Inactive
@codecov

codecov Bot commented Aug 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 66.76%. Comparing base (e431f47) to head (6494b27).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4272      +/-   ##
==========================================
- Coverage   66.79%   66.76%   -0.03%     
==========================================
  Files        1170     1170              
  Lines       44565    44565              
==========================================
- Hits        29766    29755      -11     
- Misses      14799    14810      +11     
Flag Coverage Δ
API_plugin_unit_test 68.39% <ø> (+0.02%) ⬆️
AWSPluginsCore 68.07% <ø> (ø)
Amplify 47.47% <ø> (-0.03%) ⬇️
Amplify_Foundation_Bridge_unit_test 62.28% <ø> (ø)
Amplify_Foundation_unit_test 67.64% <ø> (ø)
Analytics_plugin_unit_test 83.43% <ø> (ø)
Auth_plugin_unit_test 72.34% <ø> (-0.03%) ⬇️
Connect_unit_test 42.95% <ø> (ø)
DataStore_plugin_unit_test 81.75% <ø> (-0.10%) ⬇️
EventEnrichment_unit_test 88.98% <ø> (ø)
Firehose_plugin_unit_test 53.15% <ø> (ø)
Geo_plugin_unit_test 73.39% <ø> (ø)
Kinesis_plugin_unit_test 52.17% <ø> (ø)
Logging_plugin_unit_test 64.86% <ø> (ø)
Predictions_plugin_unit_test 34.09% <ø> (ø)
PushNotifications_plugin_unit_test 84.58% <ø> (ø)
RecordCache_unit_test 76.40% <ø> (ø)
Storage_plugin_unit_test 78.67% <ø> (ø)
unit_tests 66.76% <ø> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@harsh62
harsh62 merged commit c2814bc into main Aug 13, 2026
205 of 206 checks passed
@harsh62
harsh62 deleted the ci/resilient-dependency-resolution branch August 13, 2026 13:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants