Skip to content

Commit 9f4c2a9

Browse files
committed
rust-cache: split registry from target, prune before the target save
One entry mixed the small, lockfile-bounded registry with the huge, churning target: every lock change re-uploaded everything, the restore-key fallback accreted each generation's stale artifacts into the next, and the tar/zstd staging of a raw multi-gigabyte target has filled a runner disk at save time. The registry entry (index + tarballs + git db, toolchain-independent) keeps the restore-and-save behavior behind the existing save toggle. The target entry becomes restore-only in rust-cache; the new rust-cache-save action prunes target/ down to dependency artifacts — workspace-member objects, final binaries, examples, incremental state and docs are rebuilt or relinked on any commit — and uploads under the exact key the restore computed, handed over via GITHUB_ENV so the halves cannot diverge. It runs as the consumer's last job step, because composite actions have no post hooks; an exact restore hit skips prune and save, and save: false is the pull-request mode of the restore-everywhere/save-on-main pattern. Measured on a consumer workspace, the prune selects only member artifacts (2201 paths; dependency objects untouched) and the qa target entry is expected to drop from ~3.1 GB compressed to roughly a third. The selftest covers prune-keeps-the-dependency, prune-drops-the-member and the save: false no-op; actions/cache moves to v6 across the four uses, and the README gains the split's story plus a cache-size operations section (inventory, deleting superseded generations, clearing PR leftovers).
1 parent e07f1c7 commit 9f4c2a9

5 files changed

Lines changed: 251 additions & 43 deletions

File tree

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
name: Rust cache save
2+
description: >-
3+
The save half of `rust-cache` for the target/ entry: prune target/ down to dependency artifacts,
4+
then upload under the exact key the restore computed (handed over via $GITHUB_ENV, so the halves
5+
cannot diverge). Call it as the LAST step of the job — composite actions have no post hooks, so
6+
this is what runs "after the build, before the upload". An exact restore hit skips both prune and
7+
save (the stored entry is already pruned and current). Jobs that should only consume — pull
8+
requests, with the default branch's warmup feeding them — omit this action or pass save: false;
9+
to also save when a build step failed, put `if: always()` on this step in the workflow.
10+
11+
inputs:
12+
save:
13+
description: >-
14+
Upload the pruned target. Pass `${{ github.ref == 'refs/heads/main' }}` for the
15+
restore-everywhere/save-on-main pattern.
16+
required: false
17+
default: "true"
18+
working-directory:
19+
description: Workspace directory whose `cargo metadata` names the members to prune.
20+
required: false
21+
default: "."
22+
23+
runs:
24+
using: composite
25+
steps:
26+
# Dependency artifacts are the cache's value; everything else is rebuilt or relinked on any
27+
# commit and only grows the archive plus the tar/zstd staging that can fill the runner disk.
28+
- name: Prune target to dependency artifacts
29+
if: ${{ inputs.save == 'true' && env.RUST_CI_TARGET_KEY != '' && env.RUST_CI_TARGET_HIT != 'true' }}
30+
shell: bash
31+
working-directory: ${{ inputs.working-directory }}
32+
run: bash "$GITHUB_ACTION_PATH/prune-target.sh"
33+
34+
- name: Save target
35+
if: ${{ inputs.save == 'true' && env.RUST_CI_TARGET_KEY != '' && env.RUST_CI_TARGET_HIT != 'true' }}
36+
uses: actions/cache/save@v6
37+
with:
38+
path: ${{ env.RUST_CI_TARGET_DIR }}
39+
key: ${{ env.RUST_CI_TARGET_KEY }}
Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
#!/usr/bin/env bash
2+
# Prune target/ down to what a cache is worth keeping: DEPENDENCY artifacts.
3+
# Workspace-member objects, final binaries, examples, incremental state and
4+
# docs are rebuilt or relinked on any commit — caching them only grows the
5+
# archive and the tar/zstd staging that can fill the runner disk at save time.
6+
#
7+
# DRY_RUN=1 prints what would go instead of removing it, for local runs.
8+
set -euo pipefail
9+
10+
target="${RUST_CI_TARGET_DIR:?rust-cache did not export RUST_CI_TARGET_DIR}"
11+
if [ ! -d "$target" ]; then
12+
echo "no $target directory; nothing to prune"
13+
exit 0
14+
fi
15+
16+
remove() {
17+
if [ "${DRY_RUN:-}" = "1" ]; then
18+
printf 'would remove %s\n' "$@"
19+
else
20+
rm -rf -- "$@"
21+
fi
22+
}
23+
24+
du -sh "$target" | sed 's/^/before: /'
25+
26+
# Workspace member names; their compiled artifacts use underscores.
27+
members=$(cargo metadata --no-deps --format-version 1 | jq -r '.packages[].name' | tr '-' '_')
28+
29+
# Incremental state (defaulted off, but belt and braces), examples and docs,
30+
# in the top-level and per-triple profile directories.
31+
while IFS= read -r dir; do
32+
remove "$dir"
33+
done < <(find "$target" -mindepth 1 -maxdepth 3 -type d \
34+
\( -name incremental -o -name examples -o -name doc \) 2>/dev/null)
35+
36+
# Every profile directory: target/<profile>/ and target/<triple>/<profile>/,
37+
# recognized by their deps/ directory.
38+
for dir in "$target"/*/ "$target"/*/*/; do
39+
[ -d "${dir}deps" ] || continue
40+
# Top-level files are final binaries and their dep-info: always relinked.
41+
while IFS= read -r file; do
42+
remove "$file"
43+
done < <(find "$dir" -maxdepth 1 -type f 2>/dev/null)
44+
for name in $members; do
45+
for pattern in "${dir}deps/${name}-"* "${dir}deps/lib${name}-"* \
46+
"${dir}.fingerprint/${name}-"* "${dir}build/${name}-"*; do
47+
[ -e "$pattern" ] || continue
48+
remove "$pattern"
49+
done
50+
done
51+
done
52+
53+
du -sh "$target" | sed 's/^/after: /'

‎.github/actions/rust-cache/action.yml‎

Lines changed: 80 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,18 @@ description: >-
44
CI (no edit→recompile loop) and the main thing that balloons target/ — while OBEYING an explicit
55
CARGO_INCREMENTAL the consumer sets (some matrix/variant builds are faster with it on). It does not
66
impose a build profile: the build stays cargo's default unless the consumer sets CARGO_* vars, which
7-
flow through unchanged (incl. into the Docker actions via their CARGO_.* forwarding). Caches cargo's
8-
download cache (registry index + .crate tarballs + git db) via the first-party actions/cache;
9-
optionally caches target/ too, kept bounded by the Cargo.lock-based key (no restore-key ratchet).
7+
flow through unchanged (incl. into the Docker actions via their CARGO_.* forwarding).
8+
9+
Two independent cache entries, split by churn rate. The REGISTRY entry (index + .crate tarballs +
10+
git db, never registry/src — cargo re-extracts it, caching it is pure duplication) is small,
11+
toolchain-independent and bounded by the Cargo.lock key; it restores here and saves at job end
12+
unless `save: "false"`. The TARGET entry (opt-in via cache-target) is restore-ONLY here: a
13+
restore-key fallback plus save-on-new-key would accrete every generation's stale artifacts into
14+
the next one, so the save lives in the paired `rust-cache-save` action, which prunes target/ down
15+
to dependency artifacts first and runs as the consumer's LAST job step (composite actions have no
16+
post hooks, so an auto-save could not run after the build). Jobs that should only consume — PR
17+
jobs reading what a default-branch warmup maintains — simply omit the save step.
18+
1019
Avoids third-party cache actions whose behaviour drifts with the runner image.
1120
1221
inputs:
@@ -16,45 +25,51 @@ inputs:
1625
default: rust
1726
save:
1827
description: >-
19-
Also save the cache at job end (the actions/cache post step). Set "false" for restore-only
20-
consumers — e.g. PR jobs reading a cache that a default-branch warmup job maintains, where
21-
per-branch saves would only burn quota and job minutes.
28+
Also save the REGISTRY entry at job end (the actions/cache post step). Set "false" for
29+
restore-only consumers — e.g. PR jobs reading a cache that a default-branch warmup job
30+
maintains, where per-branch saves would only burn quota and job minutes. The target entry is
31+
restore-only here regardless; its save belongs to `rust-cache-save`.
2232
required: false
2333
default: "true"
2434
cache-target:
2535
description: >-
26-
Also cache the build target/ directory (keeps compiled dependencies between runs). Off by
36+
Also RESTORE the build target/ directory (compiled dependencies between runs). Off by
2737
default — registry/git alone is small and always safe; turn it on for a heavy dependency tree
28-
where recompiling every run is the bottleneck. Bounded by the Cargo.lock-based key (no
29-
restore-key ratchet); leave CARGO_INCREMENTAL at its default (off) to keep it tight.
38+
where recompiling every run is the bottleneck. Saving the target back is the paired
39+
`rust-cache-save` action's job (typically only on the default branch).
3040
required: false
3141
default: "false"
3242
target-dir:
33-
description: Workspace-relative target directory to cache when cache-target is true (e.g. an out-of-workspace crate's target).
43+
description: Workspace-relative target directory to restore when cache-target is true (e.g. an out-of-workspace crate's target).
3444
required: false
3545
default: target
3646
lockfiles:
37-
description: Glob of lockfile(s) whose hash keys the cache.
47+
description: Glob of lockfile(s) whose hash keys the caches.
3848
required: false
3949
default: "**/Cargo.lock"
4050
cache-directories:
41-
description: Extra newline-separated paths to add to the cache.
51+
description: >-
52+
Extra newline-separated paths to add to the registry entry (bounded, lockfile-keyed) — suited
53+
to vendored/download caches that track the lockfiles.
4254
required: false
4355
default: ""
4456

57+
outputs:
58+
target-cache-hit:
59+
description: '"true" when the target entry restored from its exact key.'
60+
value: ${{ steps.target.outputs.cache-hit }}
61+
4562
runs:
4663
using: composite
4764
steps:
4865
# Incremental compilation is pure cost in CI (no edit→recompile loop) and is what balloons target/
4966
# — gigabytes of codegen units + dep-graph data that are never reused across runs. Defaulting it
5067
# off is the single biggest size reduction and is what makes caching target/ safe; a consumer who
5168
# knows a variant build benefits can still set CARGO_INCREMENTAL and have it honoured.
52-
- name: Configure cargo for CI + compute cache key
69+
- name: Configure cargo for CI + compute cache keys
5370
id: meta
5471
shell: bash
5572
env:
56-
CACHE_TARGET: ${{ inputs.cache-target }}
57-
TARGET_DIR: ${{ inputs.target-dir }}
5873
EXTRA_DIRS: ${{ inputs.cache-directories }}
5974
run: |
6075
set -euo pipefail
@@ -63,50 +78,76 @@ runs:
6378
if [ -z "${CARGO_INCREMENTAL:-}" ]; then echo "CARGO_INCREMENTAL=0" >> "$GITHUB_ENV"; fi
6479
if [ -z "${CARGO_TERM_COLOR:-}" ]; then echo "CARGO_TERM_COLOR=always" >> "$GITHUB_ENV"; fi
6580
66-
# Fold the toolchain into the key so a rustc bump can't reuse incompatible artifacts.
81+
# Fold the toolchain into the TARGET key so a rustc bump can't reuse incompatible
82+
# artifacts. The registry (downloads only) is toolchain-independent.
6783
if command -v rustc >/dev/null 2>&1; then
6884
rust=$(rustc -V | sha256sum | cut -c1-12)
6985
else
7086
rust=norustc
7187
fi
7288
echo "rust=$rust" >> "$GITHUB_OUTPUT"
7389
74-
# Cache cargo's DOWNLOAD cache — registry index + .crate tarballs + git db. Never
75-
# registry/src: cargo re-extracts it from the tarballs, so caching it is pure duplication
76-
# (it roughly doubles the registry footprint for no benefit). The cargo home comes from
77-
# CARGO_HOME when set — container images that bake the toolchain elsewhere (e.g.
78-
# /opt/cargo) rely on it, and $HOME/.cargo would cache a directory cargo never touches.
90+
# Cargo's DOWNLOAD cache — registry index + .crate tarballs + git db. Never registry/src:
91+
# cargo re-extracts it from the tarballs, so caching it is pure duplication (it roughly
92+
# doubles the registry footprint for no benefit). The cargo home comes from CARGO_HOME when
93+
# set — container images that bake the toolchain elsewhere (e.g. /opt/cargo) rely on it,
94+
# and $HOME/.cargo would cache a directory cargo never touches.
7995
CARGO_DIR="${CARGO_HOME:-$HOME/.cargo}"
8096
{
81-
echo 'paths<<__PATHS__'
97+
echo 'registry_paths<<__PATHS__'
8298
echo "$CARGO_DIR/registry/index"
8399
echo "$CARGO_DIR/registry/cache"
84100
echo "$CARGO_DIR/git/db"
85-
if [ "$CACHE_TARGET" = "true" ]; then echo "$TARGET_DIR"; fi
86101
if [ -n "$EXTRA_DIRS" ]; then printf '%s\n' "$EXTRA_DIRS"; fi
87102
echo '__PATHS__'
88103
} >> "$GITHUB_OUTPUT"
89104
90-
# actions/cache restores here and auto-saves at job end (its post step runs even when nested in a
91-
# composite action). An exact-key save is skipped on a hit, so an unchanged Cargo.lock never grows
92-
# the cache; a changed lock seeds a fresh entry and the stale one is LRU-evicted — no ratchet.
93-
# `save: "false"` swaps in the restore-only variant of the same first-party action.
94-
- name: Cache cargo registry/git (+ target)
105+
# Registry: restores here and auto-saves at job end (its post step runs even when nested in a
106+
# composite action). An exact-key save is skipped on a hit, so an unchanged Cargo.lock never
107+
# grows the cache; a changed lock seeds a fresh entry and the stale one is LRU-evicted — no
108+
# ratchet. `save: "false"` swaps in the restore-only variant of the same first-party action.
109+
- name: Cache cargo registry/git
95110
if: inputs.save == 'true'
96-
uses: actions/cache@v5
111+
uses: actions/cache@v6
97112
with:
98-
path: ${{ steps.meta.outputs.paths }}
99-
key: ${{ inputs.prefix }}-${{ runner.os }}-rust-${{ steps.meta.outputs.rust }}-${{ hashFiles(inputs.lockfiles) }}
113+
path: ${{ steps.meta.outputs.registry_paths }}
114+
key: ${{ inputs.prefix }}-${{ runner.os }}-registry-${{ hashFiles(inputs.lockfiles) }}
100115
restore-keys: |
101-
${{ inputs.prefix }}-${{ runner.os }}-rust-${{ steps.meta.outputs.rust }}-
102-
${{ inputs.prefix }}-${{ runner.os }}-rust-
116+
${{ inputs.prefix }}-${{ runner.os }}-registry-
103117
104-
- name: Restore cargo registry/git (+ target) without saving
118+
- name: Restore cargo registry/git without saving
105119
if: inputs.save != 'true'
106-
uses: actions/cache/restore@v5
120+
uses: actions/cache/restore@v6
121+
with:
122+
path: ${{ steps.meta.outputs.registry_paths }}
123+
key: ${{ inputs.prefix }}-${{ runner.os }}-registry-${{ hashFiles(inputs.lockfiles) }}
124+
restore-keys: |
125+
${{ inputs.prefix }}-${{ runner.os }}-registry-
126+
127+
# Target: restore-only (see the action description); `rust-cache-save` prunes and uploads under
128+
# the same key, handed over via $GITHUB_ENV so the two halves can never diverge.
129+
- name: Restore target
130+
if: inputs.cache-target == 'true'
131+
id: target
132+
uses: actions/cache/restore@v6
107133
with:
108-
path: ${{ steps.meta.outputs.paths }}
109-
key: ${{ inputs.prefix }}-${{ runner.os }}-rust-${{ steps.meta.outputs.rust }}-${{ hashFiles(inputs.lockfiles) }}
134+
path: ${{ inputs.target-dir }}
135+
key: ${{ inputs.prefix }}-${{ runner.os }}-target-${{ steps.meta.outputs.rust }}-${{ hashFiles(inputs.lockfiles) }}
110136
restore-keys: |
111-
${{ inputs.prefix }}-${{ runner.os }}-rust-${{ steps.meta.outputs.rust }}-
112-
${{ inputs.prefix }}-${{ runner.os }}-rust-
137+
${{ inputs.prefix }}-${{ runner.os }}-target-${{ steps.meta.outputs.rust }}-
138+
${{ inputs.prefix }}-${{ runner.os }}-target-
139+
140+
- name: Hand the target key to rust-cache-save
141+
if: inputs.cache-target == 'true'
142+
shell: bash
143+
env:
144+
KEY: ${{ inputs.prefix }}-${{ runner.os }}-target-${{ steps.meta.outputs.rust }}-${{ hashFiles(inputs.lockfiles) }}
145+
HIT: ${{ steps.target.outputs.cache-hit }}
146+
TARGET_DIR: ${{ inputs.target-dir }}
147+
run: |
148+
set -euo pipefail
149+
{
150+
echo "RUST_CI_TARGET_KEY=$KEY"
151+
echo "RUST_CI_TARGET_DIR=$TARGET_DIR"
152+
echo "RUST_CI_TARGET_HIT=${HIT:-false}"
153+
} >> "$GITHUB_ENV"

‎.github/workflows/selftest.yml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,33 @@ jobs:
8484
fi
8585
echo "✓ malicious tool name rejected; no injection"
8686
87+
# The save half of rust-cache: prune target/ to dependency artifacts,
88+
# upload under the restore's key. Runs as (nearly) the last step, the
89+
# position consumers use; a fresh key exercises prune + upload, an
90+
# exact-hit re-run skips both (the stored entry is already pruned).
91+
- name: rust-cache-save (fixture; prune + upload)
92+
uses: ./.github/actions/rust-cache-save
93+
with:
94+
working-directory: fixtures/sample-crate
95+
96+
- name: prune kept the dependency and dropped the member
97+
if: env.RUST_CI_TARGET_HIT != 'true'
98+
run: |
99+
deps=fixtures/sample-crate/target/debug/deps
100+
ls "$deps" | grep -q '^liblibc-' || {
101+
echo "::error::dependency artifacts missing after prune"; exit 1; }
102+
if ls "$deps" | grep -q '^cicd_rust_selftest_fixture-'; then
103+
echo "::error::workspace member artifacts survived the prune"; exit 1
104+
fi
105+
echo "✓ prune kept libc, dropped the member"
106+
107+
# save: false is the pull-request mode — the step must no-op.
108+
- name: rust-cache-save honours save=false (no-op control)
109+
uses: ./.github/actions/rust-cache-save
110+
with:
111+
save: "false"
112+
working-directory: fixtures/sample-crate
113+
87114
# The Docker action set: build the image locally (no GHCR dependency to prove)
88115
# and run fetch (networked) -> lint/test (sealed, --network none) -> readiness
89116
# (networked) against the fixture.

0 commit comments

Comments
 (0)