CI: offload ccache/apt/buildx caches off the GitHub Actions cache#10685
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR restructures wolfSSL’s GitHub Actions caching to reduce cache churn/evictions and CI flakes by moving large, frequently-updated caches (apt closures, buildx layers) off the PR-scoped GitHub Actions cache and into GHCR, while making PR runs largely “read-only” for cache writes.
Changes:
- Add a scheduled/manual workflow to build and publish Ubuntu
.debbundle images to GHCR, and updateinstall-apt-depsto optionally install dependencies offline from those bundles (with apt fallback). - Update many workflows to consume GHCR
.debbundles, seed/consume shared ccache with PR read-only behavior, and switch simulator buildx caching fromtype=ghato a shared GHCR registry cache. - Reduce artifact retention for various failure log/result uploads to 7 days.
Reviewed changes
Copilot reviewed 58 out of 58 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| .github/actions/install-apt-deps/action.yml | Adds ghcr-debs-tag to install from a pulled GHCR .deb bundle; makes PR runs read-only for apt cache writes. |
| .github/actions/ccache-setup/action.yml | Adds read-only mode to restore ccache without uploading on PR runs. |
| .github/scripts/parallel-make-check.py | Adds --build-only to compile configs without running tests/commands for scheduled ccache seeding. |
| .github/workflows/ci-deps-image.yml | New scheduled/manual workflow to resolve package closures and publish .deb bundle images to GHCR. |
| .github/ci-deps/packages-ubuntu-24.04-minimal.txt | New minimal ubuntu-24.04 package manifest used to build a .deb bundle image. |
| .github/ci-deps/packages-ubuntu-24.04-full.txt | New full ubuntu-24.04 package manifest used to build a .deb bundle image. |
| .github/ci-deps/packages-ubuntu-22.04-minimal.txt | New minimal ubuntu-22.04 package manifest used to build a .deb bundle image. |
| .github/ci-deps/packages-ubuntu-22.04-full.txt | New full ubuntu-22.04 package manifest used to build a .deb bundle image. |
| .github/workflows/zephyr.yml | Uses GHCR .deb bundle tag for offline apt installs. |
| .github/workflows/wolfsm.yml | Adds weekday schedule for ccache seeding; uses GHCR .deb bundle; makes ccache read-only on PRs; uses --build-only on schedule; shortens artifact retention. |
| .github/workflows/wolfCrypt-Wconversion.yml | Adds weekday schedule for ccache seeding; uses GHCR .deb bundle; makes ccache read-only on PRs; uses --build-only on schedule; shortens artifact retention. |
| .github/workflows/trackmemory.yml | Adds weekday schedule for ccache seeding; uses GHCR .deb bundle; makes ccache read-only on PRs; uses --build-only on schedule; shortens artifact retention. |
| .github/workflows/smallStackSize.yml | Adds weekday schedule for ccache seeding; uses GHCR .deb bundle; makes ccache read-only on PRs; uses --build-only on schedule; shortens artifact retention. |
| .github/workflows/psk.yml | Adds weekday schedule for ccache seeding; uses GHCR .deb bundle; makes ccache read-only on PRs; uses --build-only on schedule; shortens artifact retention. |
| .github/workflows/pq-all.yml | Adds weekday schedule for ccache seeding; uses GHCR .deb bundle; makes ccache read-only on PRs; uses --build-only on schedule; shortens artifact retention. |
| .github/workflows/os-check.yml | Adds weekday schedule for linux ccache seeding; uses GHCR .deb bundle; switches to restore/save cache actions; adds scheduled build-only behavior; shortens artifact retention; skips Windows on schedule. |
| .github/workflows/smoke-test.yml | Adds weekday schedule for ccache seeding; uses GHCR .deb bundle; switches to restore/save cache actions; adds scheduled build-only behavior; shortens artifact retention. |
| .github/workflows/opensslcoexist.yml | Adds weekday schedule for ccache seeding; uses GHCR .deb bundle; makes ccache read-only on PRs; uses --build-only on schedule; shortens artifact retention. |
| .github/workflows/no-malloc.yml | Adds weekday schedule for ccache seeding; uses GHCR .deb bundle; makes ccache read-only on PRs; uses --build-only on schedule; shortens artifact retention. |
| .github/workflows/multi-compiler.yml | Adds weekday schedule for ccache seeding; uses GHCR .deb bundle; makes ccache read-only on PRs; uses --build-only on schedule; shortens artifact retention. |
| .github/workflows/multi-arch.yml | Adds weekday schedule for ccache seeding; uses GHCR .deb bundle; makes ccache read-only on PRs; uses --build-only on schedule; shortens artifact retention. |
| .github/workflows/cryptocb-only.yml | Adds weekday schedule for ccache seeding; uses GHCR .deb bundle; makes ccache read-only on PRs; uses --build-only on schedule; shortens artifact retention. |
| .github/workflows/disable-pk-algs.yml | Adds weekday schedule for ccache seeding; uses GHCR .deb bundle; makes ccache read-only on PRs; uses --build-only on schedule; shortens artifact retention. |
| .github/workflows/emnet-nonblock.yml | Uses GHCR .deb bundle tag for offline apt installs. |
| .github/workflows/check-headers.yml | Uses GHCR .deb bundle tag for offline apt installs. |
| .github/workflows/check-source-text.yml | Uses GHCR .deb bundle tag for offline apt installs. |
| .github/workflows/cmake.yml | Uses GHCR .deb bundle tag for offline apt installs. |
| .github/workflows/cmake-autoconf.yml | Uses GHCR .deb bundle tag for offline apt installs. |
| .github/workflows/tls-anvil.yml | Uses GHCR .deb bundle tag for offline apt installs. |
| .github/workflows/python.yml | Uses GHCR .deb bundle tag for offline apt installs. |
| .github/workflows/nss.yml | Uses GHCR .deb bundle tag for offline apt installs in both primary and fallback install steps. |
| .github/workflows/openvpn.yml | Uses GHCR .deb bundle tag for offline apt installs. |
| .github/workflows/pam-ipmi.yml | Uses GHCR .deb bundle tag for offline apt installs. |
| .github/workflows/rng-tools.yml | Uses GHCR .deb bundle tag for offline apt installs. |
| .github/workflows/softhsm.yml | Uses GHCR .deb bundle tag for offline apt installs. |
| .github/workflows/socat.yml | Uses GHCR .deb bundle tag for offline apt installs. |
| .github/workflows/jwt-cpp.yml | Uses GHCR .deb bundle tag for offline apt installs. |
| .github/workflows/ipmitool.yml | Uses GHCR .deb bundle tag for offline apt installs. |
| .github/workflows/haproxy.yml | Uses GHCR .deb bundle tag for offline apt installs. |
| .github/workflows/grpc.yml | Uses GHCR .deb bundle tag for offline apt installs. |
| .github/workflows/msmtp.yml | Uses GHCR .deb bundle tag for offline apt installs. |
| .github/workflows/mosquitto.yml | Uses GHCR .deb bundle tag for offline apt installs. |
| .github/workflows/mono.yml | Uses GHCR .deb bundle tag for offline apt installs. |
| .github/workflows/memcached.yml | Uses GHCR .deb bundle tag for offline apt installs. |
| .github/workflows/curl.yml | Uses GHCR .deb bundle tag for offline apt installs. |
| .github/workflows/cyrus-sasl.yml | Uses GHCR .deb bundle tag for offline apt installs. |
| .github/workflows/bind.yml | Uses GHCR .deb bundle tag for offline apt installs. |
| .github/workflows/ada.yml | Uses GHCR .deb bundle tag for offline apt installs. |
| .github/workflows/atecc608-sim.yml | Switches buildx caching to GHCR registry cache; adds schedule/dispatch writers and GHCR login; adds permissions for package writes. |
| .github/workflows/se050-sim.yml | Switches buildx caching to GHCR registry cache; adds schedule/dispatch writers and GHCR login; adds permissions for package writes. |
| .github/workflows/stsafe-a120-sim.yml | Switches buildx caching to GHCR registry cache; adds schedule/dispatch writers and GHCR login; adds permissions for package writes. |
| .github/workflows/stm32-sim.yml | Switches buildx caching to GHCR registry cache (per-chip tags); adds schedule/dispatch writers and GHCR login; adds permissions for package writes. |
| .github/workflows/tropic01-sim.yml | Switches buildx caching to GHCR registry cache; adds schedule/dispatch writers and GHCR login; adds permissions for package writes. |
| .github/workflows/pic32mz-sim.yml | Switches buildx caching to GHCR registry cache; adds schedule/dispatch writers and GHCR login; adds permissions for package writes. |
| .github/workflows/docker-OpenWrt.yml | Switches buildx caching to GHCR registry cache; adds schedule/dispatch writers and GHCR login; adds permissions for package writes. |
| .github/workflows/wolfboot-integration.yml | Caps uploaded test result artifact retention to 7 days. |
| .github/workflows/watcomc.yml | Caps uploaded artifact retention to 7 days. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
retest this please |
dgarske
reviewed
Jun 15, 2026
dgarske
left a comment
dgarske
left a comment
Member
There was a problem hiding this comment.
Skoll Code Review
Scan type: reviewOverall recommendation: COMMENT
Findings: 5 total — 5 posted, 0 skipped
5 finding(s) posted as inline comments (see file-level comments below)
Posted findings
- [Medium] Scheduled ccache reseed (CCACHE_RECACHE) does not cover the os-check linux shards —
.github/workflows/os-check.yml:95-106,401-406 - [Medium] release/ pushes no longer refresh the os-check linux ccache** —
.github/workflows/os-check.yml:95-96,401-406 - [Low] No CI guard keeping consumer 'packages' a subset of the ghcr bundle lists —
.github/ci-deps/packages-ubuntu-24.04-minimal.txt:1-20 - [Low] Dead run_id/run_attempt suffix in the read-only ccache restore key —
.github/actions/ccache-setup/action.yml:70-78 - [Low] Inconsistent ghcr owner reference between install-apt-deps and sim workflows —
.github/actions/install-apt-deps/action.yml:50-51
Review generated by Skoll
julek-wolfssl
force-pushed
the
ci-cache-offload
branch
from
9d9feb5 to
9c57fa8
Compare
The 10 GB, LRU-evicted, PR-scoped Actions cache was being thrashed - the
docker simulator buildx layers (~6 GiB), plus per-PR ccache and apt-archive
writes whose keys never hit - which kept evicting the shared ccache, while
the apt mirror timed out often enough to break PR CI. Move the heavy caches
to ghcr (free, separate pool) and make PR runs read-only against the Actions
cache.
apt dependencies from prebuilt ghcr .deb bundles
- ci-deps-image.yml resolves each package list under .github/ci-deps/ into
its .deb closure and publishes ghcr.io/<owner>/wolfssl-ci-debs:<tag> in
two tiers: <ver>-minimal (make-check family) and <ver>-full (interop
superset), for ubuntu-22.04 and 24.04.
- install-apt-deps gains a ghcr-debs-tag input: pull the bundle and install
offline (--no-download) so the apt mirror is never on the PR critical
path. Any failure (bundle missing/not public/incomplete) falls through to
the existing apt path, so it is always safe to set.
sim-test buildx layers to a shared ghcr registry cache
- the 7 docker simulator workflows switch from cache-to: type=gha to
ghcr.io/wolfssl/wolfssl-sim-cache:<scope>. cache-from reads on every run
(anonymous); cache-to writes only on the weekend cron and manual
workflow_dispatch. Per-distinct-image tags and de-duplicated writers keep
parallel matrix jobs from racing on one ref.
ccache: PRs read, the schedule writes
- ccache-setup gains read-only: PR runs restore the shared master-scoped
cache but never upload; schedule/push runs refresh it. Wired across
os-check (linux + macOS), pq-all, smoke-test and the 12 small make-check
workflows.
- parallel-make-check.py gains --build-only (compile every config, skip the
test phase) so weekday-morning seed crons warm the cache PR runs consume.
artifact retention capped at 7 days on the failure-log/result uploads that
previously defaulted to 90.
ONE-TIME SETUP: after their first publish, make the ghcr packages
wolfssl-ci-debs and wolfssl-sim-cache PUBLIC so anonymous pulls work from PR
(including fork) runs; until then everything falls back cleanly.
…ments - install-apt-deps: the ghcr offline-install path now honors the no-install-recommends input; it was always installing recommends, diverging from the regular apt path. - install-apt-deps: correct the ghcr-debs-tag example to a real tag (ubuntu-24.04-minimal) - ci-deps-image publishes -minimal/-full variants, not a bare <ver> tag. - os-check: fix the schedule header comment - macOS runs --build-only on the weekday cron to seed its ccache (like the linux shards); only Windows is skipped on schedule.
The scheduled (cron) refresh restored the prior ccache and recompiled only the translation units that changed, so unchanged objects were never rebuilt and the shared cache could drift indefinitely. Set CCACHE_RECACHE=1 on schedule events - gated inside the ccache-setup action, so none of the calling workflows change - to force fresh compiles that re-store every result. PR and push runs are unaffected and keep their warm hits; only the scheduled jobs pay the full recompile.
The scheduled-reseed step gated CCACHE_RECACHE with
`if: github.event_name == 'schedule'`, but the github context is not
available in a composite action's step-level if:. The action manifest
therefore failed to load ("Unrecognized named-value: 'github'"), and
every workflow using ccache-setup broke at the "Set up ccache" step
(build library, make check, Compiler test, Multi-arch test, ...).
Gate on the built-in $GITHUB_EVENT_NAME env var in the shell instead,
which keeps the schedule-only reseed behaviour with no caller changes.
The read-only input description embedded `${{ github.event_name ==
'pull_request' }}` as example text. GitHub validates ${{ }} expressions in
an action's input definitions at manifest-load time, where the github
context is not available, so the action failed to load ("Unrecognized
named-value: 'github'", action.yml line 27) and every ccache-setup
consumer died at "Set up ccache" (build library, make check, Compiler
test, Multi-arch test, ...). Describe the expression in prose instead of
embedding it as a live ${{ }} template; the github.* references that
remain are in step with:/run: blocks, where the context is available.
- ccache-setup: gate the scheduled-reseed step with
`if: github.event_name == 'schedule'` again. The github context IS
available in a composite action's step-level if: (install-apt-deps
already relies on it), so the earlier $GITHUB_EVENT_NAME workaround and
its comment were based on a wrong premise. The real load failure was the
${{ }} expression in the read-only input description, fixed separately.
- smoke-test.yml: include github.event_name in the concurrency group. The
workflow pushes to master/main and now also runs on a weekday schedule;
both share github.ref on the default branch, so under
cancel-in-progress a seed run and a master push could cancel each other.
Tighten three pieces of documentation to match the implementation; no behaviour change: - install-apt-deps (ghcr-debs-tag description): the apt mirror is avoided only on the successful offline path. The offline install is a single --no-download install of the whole package set, so any miss (bundle absent/private/incomplete) falls back to the apt path. - ci-deps-image header: each bundle is every requested package plus the dependencies not already present on the matching runner image - tied to that runner, not a portable/self-contained .deb closure. - ci-deps-image schedule note: a package missing from the bundle fails the whole offline install (it is not per-package), falling back to the full apt path.
- os-check.yml linux shard: add a schedule-gated CCACHE_RECACHE=1 step so the weekday seed reseeds from clean compiles rather than only accumulating deltas. This shard manages ccache directly (its own restore/save) and so was not covered by the ccache-setup composite's reseed. - install-apt-deps: hardcode the ghcr bundle owner to wolfssl. The bundle is only published under ghcr.io/wolfssl by ci-deps-image, so fork PRs now read the public upstream image instead of a nonexistent ghcr.io/<fork>/wolfssl-ci-debs. - ccache-setup: document that the read-only restore key reuses the save key shape for symmetry and is never an exact hit by design. Skoll F3 (a packages-subset-of-bundle CI guard) is deferred to a follow-up; F4 (release-branch ccache saves) is left as the intended seed-on-schedule / everything-else-reads model.
Rebasing onto master (which migrated JS actions to Node.js 24 runtimes) left a few action refs that this branch added in new steps still on the old major versions. Bring them in line with master: - ccache-setup read-only restore: actions/cache/restore@v4 -> @v5 - smoke-test / os-check ccache save: actions/cache/save@v4 -> @v5 - ci-deps-image checkout: actions/checkout@v4 -> @v5
julek-wolfssl
force-pushed
the
ci-cache-offload
branch
from
9c57fa8 to
634ac9b
Compare
dgarske
approved these changes
Jun 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The 10 GB, LRU-evicted, PR-scoped GitHub Actions cache was being thrashed: the
docker simulator buildx layers (~6 GiB), plus per-PR ccache and apt-archive
writes whose keys never hit, kept evicting the shared ccache, while the apt
mirror timed out often enough to break PR CI.
This moves the heavy caches to ghcr (free, separate pool) and makes PR runs
read-only against the Actions cache.
Changes
apt dependencies from prebuilt ghcr
.debbundlesci-deps-image.ymlresolves each package list under.github/ci-deps/intoits
.debclosure and publishesghcr.io/<owner>/wolfssl-ci-debs:<tag>intwo tiers:
<ver>-minimal(make-check family) and<ver>-full(interopsuperset), for ubuntu-22.04 and 24.04.
install-apt-depsgains aghcr-debs-taginput: pull the bundle and installoffline (
--no-download) so the apt mirror is never on the PR critical path.Any failure (bundle missing/not public/incomplete) falls through to the
existing apt path, so it is always safe to set.
sim-test buildx layers to a shared ghcr registry cache
cache-to: type=ghatoghcr.io/wolfssl/wolfssl-sim-cache:<scope>.cache-fromreads on every run(anonymous);
cache-towrites only on the weekend cron and manualworkflow_dispatch. Per-distinct-image tags and de-duplicated writers keepparallel matrix jobs from racing on one ref.
ccache: PRs read, the schedule writes
ccache-setupgainsread-only: PR runs restore the shared master-scopedcache but never upload; schedule/push runs refresh it. Wired across os-check
(linux + macOS), pq-all, smoke-test and the 12 small make-check workflows.
parallel-make-check.pygains--build-only(compile every config, skip thetest phase) so weekday-morning seed crons warm the cache PR runs consume.
Misc
previously defaulted to 90.
After their first publish, make the ghcr packages
wolfssl-ci-debsandwolfssl-sim-cachepublic so anonymous pulls work from PR (including fork)runs. Until then everything falls back cleanly.