ci: seed the SwiftPM dependencies cache and raise job timeouts by harsh62 · Pull Request #4269 · aws-amplify/amplify-swift

harsh62 · 2026-08-11T16:25:15Z

Issue

CI jobs are being killed at their timeout with no build output at all. Job logs end after a few lines and cleanup terminates orphaned git and git-remote-http processes — the jobs were still cloning dependencies, not compiling or testing.

Root cause is that the shared SwiftPM dependencies cache has never populated. gh api .../actions/caches reports total_count: 0.

Every unit test, integration test, and build job only ever restores amplify-packages-*. The single step that writes it lives in build_scheme.yml gated on github.ref_name == 'main' — but the only workflow that reaches that step with saving enabled is build_amplify_swift_platforms.yml, which declares branches-ignore: [main] and is otherwise only reachable from the deploy workflows that run on release. The write is unreachable.

So all ~190 jobs on every run independently clone the full dependency graph (~34 packages: aws-sdk-swift, smithy-swift, aws-crt-swift, swift-nio, …) and starve each other.

Description

1. Seed the cache from `main` (new workflow)

seed_dependencies_cache.yml resolves dependencies on main and saves the cache, so PR jobs restore instead of cloning. It runs only when Package.resolved/Package.swift change, and its concurrency group prevents two seeding runs racing the same key.

It prunes the bare git mirrors under repositories/ before saving: only checkouts/ is needed once working copies exist, and consumers build with -disableAutomaticPackageResolution on a cache hit. That takes the entry from ~4.3 GB to ~830 MB, which matters against the repository-wide 10 GB cache budget.

2. Drop the Xcode version from the cache key

- key: amplify-packages-${{ steps.platform.outputs.xcode-version }}-${{ hashFiles('Package.resolved') }}
+ key: amplify-packages-${{ hashFiles('Package.resolved') }}

These checkouts are plain source and not toolchain-specific — Package.resolved already pins them exactly. Keying on Xcode meant every Xcode bump invalidated the entire cache at once, including via restore-keys. Applied to all five workflows that share the key so they keep sharing it.

3. Raise timeouts 30 → 45 minutes

Including the nine callers that hardcoded 30 and so overrode the reusable-workflow default. The cache is only seeded from main by design, so PR runs must tolerate a cold start.

4. Run the Xcode preview build on PRs

Moves the Xcode 27 preview build from a nightly schedule to pull_request + push: main, so next-Xcode breakages surface on the PR that introduces them rather than on a run nobody is watching. Every job keeps continue-on-error — a beta toolchain on a capacity-constrained preview image is not a suitable merge gate.

Why this is a separate PR

This was found while working on the minimum-version bump (#4268), but it is an independent pre-existing bug and that PR cannot go green until the cache is seeded from main. Splitting it breaks the circular dependency and keeps both diffs reviewable. This PR touches only .github/.

How did you test these changes?

Verified the cache is empty (total_count: 0) and that build_amplify_swift_platforms.yml's branches-ignore: [main] makes the existing save step unreachable.
Reproduced the stall locally with an empty clone directory and sampled the stalled process. The stack confirms the mechanism:
```
Workspace.loadPackageGraph → Workspace._resolve
  → Workspace.updateDependenciesCheckouts → Workspace.checkoutRepository
    → GitRepository.checkout(revision:) → AsyncProcess.waitUntilExit()
```
xcodebuild is blocked in SwiftPM waiting on git subprocesses checking out working copies — not simulator boot, not compilation. SwiftPM emits no progress during this phase, which is why the jobs look hung.
Confirmed the local run completes and passes when given enough time (** TEST SUCCEEDED **), with the tests themselves taking 3.4 seconds.
Verified xcodebuild -resolvePackageDependencies populates the cache directory with 31 checkouts, that resolution still succeeds after pruning repositories/, and that a real consumer build (AWSPredictionsPlugin for macOS) succeeds against the pruned directory with -disableAutomaticPackageResolution.
All 77 workflow/action YAML files parse.

The seeding workflow cannot run until it is on main — that is the point of this PR. The first main run after merge will populate the cache; subsequent PRs get a warm start.

Documentation

No customer-facing changes — CI configuration only.

Checklist

PR description included
Unit tests added/updated — not applicable; CI configuration change, validated by YAML parsing and by locally reproducing and diagnosing the stall
Documentation updated — inline comments explain why the key omits the Xcode version and why repositories/ is pruned

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Jobs on this PR were being killed at exactly 30 minutes, scattered randomly across schemes and platforms rather than clustered on any one target. The job logs show no build output at all before the kill, and cleanup terminating orphaned git and git-remote-http processes: the jobs were still cloning dependencies, not compiling. Root cause is the dependencies cache key. It embedded the Xcode version: amplify-packages-${{ steps.platform.outputs.xcode-version }}-<Package.resolved hash> Raising the latest Xcode from 26.3 to 26.5 made every existing entry unreachable, including via restore-keys. With no entry to restore, the build cache restore is skipped too (it is gated on a dependencies cache hit), so xcodebuild ran against an empty clonedSourcePackagesDirPath and had to clone the full dependency graph (~34 packages: aws-sdk-swift, smithy-swift, aws-crt-swift, swift-nio, ...). Dozens of jobs did that concurrently and starved each other until they timed out. These checkouts are just source and are not toolchain specific -- Package.resolved already pins them exactly -- so the Xcode version is dropped from the key. Otherwise this breaks again on every Xcode bump. Applied to all five workflows that share the key so they keep sharing it. Timeouts raised from 30 to 45 minutes, including the nine callers that hardcoded 30 and so overrode the reusable workflow default. A cold run legitimately needs more than 30 minutes, and the cache is only seeded from main by design, so PR runs must tolerate a cold start. Note the dependencies cache is intentionally only written on main (build_scheme.yml), so PRs stay consumers and cannot poison it.

Runs the Xcode 27 preview build on PRs and pushes to main instead of on a nightly schedule, so breakages from the next Xcode surface on the PR that introduces them rather than hours later on a run nobody is watching. Every job keeps continue-on-error, so this reports without gating a merge. The preview image runs a beta toolchain and can have constrained capacity, which makes it unsuitable as a required check. Renamed from nightly_xcode_preview.yml to build_xcode_preview.yml to match the trigger, and the concurrency group now keys on the PR number like the other PR workflows so superseded runs are cancelled.

Raising the job timeout from 30 to 45 minutes did not fix the killed jobs; it only moved the wall. Jobs still die with no build output, and passing jobs on the same run cluster at 28-43 minutes, right up against the new ceiling. That is contention, not a slow step, and no timeout value fixes it. The dependencies cache never populates. Every unit test, integration test and build job only *restores* `amplify-packages-*`. The single step that writes it lives in build_scheme.yml gated on `github.ref_name == 'main'`, but the only workflow that reaches it with save_build_cache enabled is build_amplify_swift_platforms.yml, which declares `branches-ignore: [main]` and is otherwise only reachable from the deploy workflows that run on release. The write was unreachable, so the cache has always been empty (the API reports total_count 0). The result is that all ~190 jobs on a run each clone the full dependency graph (~34 packages) from github.com concurrently and starve each other. A clean resolve takes about 5 minutes locally, so the 45 minutes of silence is contention between jobs, not the resolve itself. This adds a workflow that resolves dependencies on main and saves the cache, so PR jobs restore it instead of cloning. It runs only when Package.resolved or Package.swift change, and its concurrency group prevents two seeding runs from racing the same key. The bare git mirrors under repositories/ are pruned before saving: only checkouts/ is needed once working copies exist, and consumers build with -disableAutomaticPackageResolution on a cache hit. That takes the entry from ~4.3GB to ~830MB, which matters against the repository-wide 10GB budget. Verified locally that resolving into the cache directory produces 31 checkouts, that resolution still succeeds after pruning repositories/, and that a real consumer build (AWSPredictionsPlugin for macOS) succeeds against the pruned directory with -disableAutomaticPackageResolution.

The seeding job has to resolve dependencies cold, because it exists precisely because no cache is available yet. It is also the single job every other workflow depends on for a warm start, so a timeout there leaves the cache unseeded and every PR job cold-cloning again. It runs alone rather than alongside ~190 siblings, so it should finish in minutes. Raised from 45 to 90 minutes purely so contention cannot leave the cache unseeded.

Two gaps meant a restored cache could still be ignored, leaving jobs to cold-clone the dependency graph and hit the job timeout anyway. The restore step had timeout-minutes: 4 with continue-on-error: true. The entry is ~830MB and many jobs download it at once, so a slow download exceeds 4 minutes, the step is silently abandoned, and the job proceeds with no cache -- exactly the failure this is meant to prevent. Raised to 10 minutes. More importantly, resolution was gated on cache-hit, which is only 'true' on an exact key match. A restore-keys prefix match reports an empty cache-hit, so a job that successfully restored usable checkouts would still re-resolve and re-clone the whole graph. Switched to cache-matched-key != '', which is set for both exact and prefix matches. Applied to all five workflows that share the key. The save gates deliberately keep cache-hit != 'true', since those should only write when the exact key was absent. Verified locally that the degraded path is safe: building against a cache with repositories/ pruned and resolution left enabled took 1m32s versus 2m48s with no cache at all, fetched nothing, and did not recreate the pruned mirrors (** BUILD SUCCEEDED **). Re-resolution alone against that cache took 36s. So a prefix match is genuinely usable rather than a trap.

Not for merge. A workflow is not dispatchable until it exists on the default branch, so the only way to seed the cache and measure whether it actually prevents the cold-clone stall is to let the push trigger fire here. Reverted before merge.

Two defects, both of which meant the cache could never work and jobs could hang until the job timeout with no output. 1. The cache key could change within a single job. Keys hashed Package.resolved. But `xcodebuild -resolvePackageDependencies` rewrites that file when the resolving toolchain writes a different format version: Xcode 16 writes version 2, Xcode 26 / Swift 6 writes version 3 with an originHash. The seeding job proved it — the restore step asked for amplify-packages-4ebbfe5... and the save step, in the same job, wrote amplify-packages-0c76351... So the saved key was one no consumer would ever request, and the cache could never hit. Keys now hash Package.swift, which resolution does not rewrite. Applied to all 12 key sites (7 dependency, 5 build cache) so they stay in lockstep. 2. Resolution could hang silently inside xcodebuild. Left to xcodebuild, SwiftPM checks out ~34 repositories one at a time via blocking git subprocesses and prints nothing until the first compile. A slow clone therefore looked like a hang and was killed at the job timeout, taking the test run with it — the observed failures had zero build output and orphaned git / git-remote-http processes. Resolution is now an explicit step in both run_xcodebuild and run_xcodebuild_test, bounded at 20 minutes on its own clock, logged, and retried once after clearing partial mirror state. If it genuinely cannot resolve it fails fast with a clear reason instead of consuming the whole job budget. The build and test invocations now always pass -disableAutomaticPackageResolution, so whether the checkouts came from the cache or from that step, xcodebuild can never fall back to resolving mid-build. Net effect: the cold path is bounded and fails loudly, the warm path skips resolution entirely, and neither can silently exhaust the job timeout. Verified locally on a fully cold cache: explicit resolve took 2m0s and produced 31 checkouts, then `xcodebuild test` with resolution disabled took 1m41s and passed (** TEST SUCCEEDED **, 31 tests). Under 4 minutes total, against 30-45 minute hangs before.

Empty commit to measure the warm path: whether jobs restore the seeded 144MB cache, skip resolution, and finish well inside the timeout.

The previous commit put `timeout-minutes:` on a step inside a composite action. GitHub does not allow that, so every job using these actions failed immediately with: Unexpected value 'timeout-minutes' Failed to load .../run_xcodebuild_test/action.yml Replaced with a background watchdog that kills the resolve after 1200s. `timeout(1)` is not available on macOS runners, so the bound is implemented with a sleep-and-kill subshell, and the watchdog is reaped afterwards so it does not emit job-control noise. Verified the watchdog locally across all three paths: fast success returns 0, fast failure returns non-zero, and a hang is killed at the bound (3s in the test harness) rather than running to completion.

harsh62 · 2026-08-12T22:43:54Z

Superseded. The cache-seeding approach is being dropped: seeding only works once the workflow is on the default branch, and GitHub scopes branch caches so it could never help sibling PRs. Replacing it with a branch that makes dependency resolution resilient whether or not a cache exists.

harsh62 added 3 commits August 11, 2026 12:19

harsh62 requested a review from a team as a code owner August 11, 2026 16:25

harsh62 had a problem deploying to IntegrationTest August 11, 2026 16:25 — with GitHub Actions Error

harsh62 temporarily deployed to IntegrationTest August 11, 2026 16:25 — with GitHub Actions Inactive

harsh62 had a problem deploying to IntegrationTest August 11, 2026 16:25 — with GitHub Actions Error

harsh62 temporarily deployed to IntegrationTest August 11, 2026 16:25 — with GitHub Actions Inactive

harsh62 had a problem deploying to IntegrationTest August 11, 2026 16:25 — with GitHub Actions Error

harsh62 temporarily deployed to IntegrationTest August 11, 2026 16:25 — with GitHub Actions Inactive

harsh62 had a problem deploying to IntegrationTest August 11, 2026 16:25 — with GitHub Actions Error

harsh62 temporarily deployed to IntegrationTest August 11, 2026 16:25 — with GitHub Actions Inactive

harsh62 deployed to Fortify August 11, 2026 18:08 — with GitHub Actions Active

harsh62 temporarily deployed to IntegrationTest August 11, 2026 18:08 — with GitHub Actions Inactive

harsh62 mentioned this pull request Aug 11, 2026

chore: raise minimum supported platform, Swift, and Xcode versions #4271

Merged

8 tasks

harsh62 added 4 commits August 12, 2026 13:50

ci: trigger a run against the now-populated dependency cache

0e872d5

Empty commit to measure the warm path: whether jobs restore the seeded 144MB cache, skip resolution, and finish well inside the timeout.

Navigation Menu

Uh oh!

ci: seed the SwiftPM dependencies cache and raise job timeouts - #4269

ci: seed the SwiftPM dependencies cache and raise job timeouts#4269
harsh62 wants to merge 9 commits into
mainfrom
ci/dependency-cache-and-timeouts

harsh62 commented Aug 11, 2026

Uh oh!

harsh62 commented Aug 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

harsh62 commented Aug 11, 2026

Issue

Description

1. Seed the cache from main (new workflow)

2. Drop the Xcode version from the cache key

3. Raise timeouts 30 → 45 minutes

4. Run the Xcode preview build on PRs

Why this is a separate PR

How did you test these changes?

Documentation

Checklist

Uh oh!

harsh62 commented Aug 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

1. Seed the cache from `main` (new workflow)