Skip to content

[ci] flatten caches #3999

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
andresilva wants to merge 8 commits into
base: main
Choose a base branch
from
Open

[ci] flatten caches #3999

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
5554901
[ci] flatten caches
andresilva Jun 9, 2026
de3cbf5
[ci] remove preinstalled rust toolchains
andresilva Jun 9, 2026
6c4e8bd
[ci] temporarily save cache on PR
andresilva Jun 9, 2026
fee5f9e
Revert "[ci] temporarily save cache on PR"
andresilva Jun 9, 2026
4d9a7b4
[ci] setup handle empty toolchains
andresilva Jun 10, 2026
327b252
[ci] add llvm-tools-preview component for coverage job
andresilva Jun 10, 2026
be57207
[ci] harden key normalization
andresilva Jun 10, 2026
d2ea0d7
[ci] setup nits
andresilva Jun 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 51 additions & 7 deletions .github/actions/setup/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,17 @@ inputs:
required: false
default: ""
additional-cache-key:
description: "An additional cache key that is added alongside the automatic `job`-based cache key."
description: "An additional cache key appended to the workflow-name prefix and automatic job-based cache key."
required: false
default: ""
rust-cache-enabled:
description: "Whether rust-cache should restore and save cache entries for this job."
required: false
default: "true"
rust-cache-save:
description: "Whether this job should save rust-cache entries when running on main."
required: false
default: "true"
runs:
using: composite
steps:
Expand All @@ -45,6 +53,22 @@ runs:
echo "CARGO_TARGET_DIR=/mnt/target" >> "$GITHUB_ENV"
echo "CARGO_HOME=/mnt/.cargo" >> "$GITHUB_ENV"
echo "/mnt/.cargo/bin" >> "$GITHUB_PATH"
- name: Remove preinstalled Rust toolchains
if: inputs.rust-cache-enabled == 'true'
shell: bash
run: |
# rust-cache hashes every rustup toolchain it can see. Hosted runners
# ship a rolling stable toolchain, so start from an empty toolchain set.
toolchains="$(rustup toolchain list --quiet)"
while IFS= read -r line; do
if [ "$line" = "no installed toolchains" ]; then
continue
fi
toolchain="${line%% *}"
if [ -n "$toolchain" ]; then
rustup toolchain uninstall "$toolchain"
fi
done <<< "$toolchains"
- name: Install additional Rust toolchain
if: inputs.additional-toolchain != ''
uses: dtolnay/rust-toolchain@f7ccc83f9ed1e5b9c81d8a67d7ad1a747e22a561
Expand All @@ -59,19 +83,39 @@ runs:
components: ${{ inputs.components }}
targets: ${{ inputs.targets }}
- name: Normalize cache key
if: inputs.rust-cache-enabled == 'true'
id: cache-key
shell: bash
env:
WORKFLOW_NAME: ${{ github.workflow }}
ADDITIONAL_CACHE_KEY: ${{ inputs.additional-cache-key }}
run: |
key="${{ inputs.additional-cache-key }}"
key="${key// /-}"
key="${key//\//-}"
key="${WORKFLOW_NAME}"
if [ -n "$ADDITIONAL_CACHE_KEY" ]; then
key="${key}-${ADDITIONAL_CACHE_KEY}"
fi
key="$(printf '%s' "$key" | LC_ALL=C tr -c 'A-Za-z0-9._-' '-')"
echo "key=${key}" >> "$GITHUB_OUTPUT"
- name: Configure rust cache workspace
if: inputs.rust-cache-enabled == 'true'
id: rust-cache-workspace
shell: bash
run: |
target="${CARGO_TARGET_DIR:-target}"
# rust-cache's workspace mapping is written as `. -> target`. On linux,
# cargo uses /mnt/target for disk space, so convert it back to a path
# relative to GITHUB_WORKSPACE before passing it to rust-cache
if [ "${RUNNER_OS:-}" = "Linux" ] && [ "${target#/}" != "$target" ]; then
target="$(realpath --relative-to="$GITHUB_WORKSPACE" "$target")"
fi
echo "target=${target}" >> "$GITHUB_OUTPUT"
- name: Setup rust cache
if: inputs.rust-cache-enabled == 'true'
uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1
with:
save-if: ${{ github.ref == 'refs/heads/main' }}
save-if: ${{ github.ref == 'refs/heads/main' && inputs.rust-cache-save == 'true' }}
key: ${{ steps.cache-key.outputs.key }}
cache-on-failure: false
cache-workspace-crates: true
cache-directories: ${{ env.CARGO_TARGET_DIR }}
cache-workspace-crates: false
workspaces: . -> ${{ steps.rust-cache-workspace.outputs.target }}
cache-bin: true
2 changes: 2 additions & 0 deletions .github/workflows/benchmark.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,8 @@ jobs:
- uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4
- name: Run setup
uses: ./.github/actions/setup
with:
additional-cache-key: ${{ matrix.package }}-${{ matrix.cargo_flags || 'default' }}
- name: Compile benchmarks
env:
RUSTFLAGS: "--cfg full_bench"
Expand Down
4 changes: 3 additions & 1 deletion .github/workflows/coverage.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,9 @@ jobs:
uses: ./.github/actions/setup
with:
additional-toolchain: ${{ env.NIGHTLY_VERSION }}
additional-cache-key: ${{ matrix.flags || 'default' }}-part-${{ matrix.partition }}
additional-components: llvm-tools-preview
additional-cache-key: ${{ matrix.flags || 'default' }}
rust-cache-save: ${{ matrix.partition == 1 }}

@patrick-ogrady patrick-ogrady Jun 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This assumes that crates are largely shared by partitions? when I've poked around, I find they differ a decent bit? Maybe that is a fluke?

@andresilva andresilva Jun 9, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rust-cache is only caching dependencies, it doesn't cache anything from the workspace

- name: Remove fuzz
run: |
rm -rf broadcast/fuzz
Expand Down
14 changes: 10 additions & 4 deletions .github/workflows/fast.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ jobs:
- name: Run setup
uses: ./.github/actions/setup
with:
components: clippy
additional-toolchain: ${{ env.NIGHTLY_VERSION }}
additional-components: ${{ matrix.os == 'ubuntu-latest' && matrix.flags == '' && 'rustfmt,rustc-dev,llvm-tools-preview' || 'rustfmt' }}
additional-cache-key: ${{ matrix.flags || 'default' }}
Expand Down Expand Up @@ -103,7 +104,8 @@ jobs:
- name: Run setup
uses: ./.github/actions/setup
with:
additional-cache-key: ${{ matrix.flags || 'default' }}-part-${{ matrix.partition }}
additional-cache-key: ${{ matrix.flags || 'default' }}
rust-cache-save: ${{ matrix.partition == 1 }}
- name: Install just & nextest
uses: taiki-e/install-action@0c5db7f7f897c03b771660e91d065338615679f4 # v2.60.0
with:
Expand Down Expand Up @@ -158,7 +160,7 @@ jobs:
- name: Run setup
uses: ./.github/actions/setup
with:
additional-cache-key: stability-${{ matrix.level }}
additional-cache-key: ${{ matrix.level }}
- name: Install just
uses: taiki-e/install-action@0c5db7f7f897c03b771660e91d065338615679f4 # v2.60.0
with:
Expand All @@ -179,7 +181,6 @@ jobs:
uses: ./.github/actions/setup
with:
additional-toolchain: ${{ env.NIGHTLY_VERSION }}
additional-cache-key: unstable-public
- name: Install just
uses: taiki-e/install-action@0c5db7f7f897c03b771660e91d065338615679f4 # v2.60.0
with:
Expand Down Expand Up @@ -238,6 +239,8 @@ jobs:
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4
- name: Run setup
uses: ./.github/actions/setup
with:
rust-cache-enabled: false
- name: Install just & cargo-cooldown
uses: taiki-e/install-action@0c5db7f7f897c03b771660e91d065338615679f4 # v2.60.0
with:
Expand Down Expand Up @@ -323,6 +326,8 @@ jobs:
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4
- name: Run setup
uses: ./.github/actions/setup
with:
rust-cache-enabled: false
- name: Check no_std compatibility
run: ./.github/scripts/check_no_std.sh

Expand Down Expand Up @@ -351,6 +356,8 @@ jobs:
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4
- name: Run setup
uses: ./.github/actions/setup
with:
rust-cache-enabled: false
- name: Check publish order
run: python3 .github/scripts/check_publish_order.py

Expand Down Expand Up @@ -398,7 +405,6 @@ jobs:
uses: ./.github/actions/setup
with:
toolchain: ${{ steps.toolchain.outputs.version }}
additional-cache-key: toolchain-${{ matrix.toolchain }}
- name: Install just & cargo-hack
uses: taiki-e/install-action@0c5db7f7f897c03b771660e91d065338615679f4 # v2.60.0
with:
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/loom.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,8 @@ jobs:
- name: Run setup
uses: ./.github/actions/setup
with:
additional-cache-key: loom-${{ matrix.flags || 'default' }}-part-${{ matrix.partition }}
additional-cache-key: ${{ matrix.flags || 'default' }}
rust-cache-save: ${{ startsWith(matrix.partition, '1/') }}
- name: Install just & nextest
uses: taiki-e/install-action@0c5db7f7f897c03b771660e91d065338615679f4 # v2.60.0
with:
Expand Down
12 changes: 9 additions & 3 deletions .github/workflows/slow.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,8 @@ jobs:
- name: Run setup
uses: ./.github/actions/setup
with:
additional-cache-key: ${{ matrix.flags || 'default' }}-part-${{ matrix.partition }}
additional-cache-key: ${{ matrix.flags || 'default' }}
rust-cache-save: ${{ matrix.partition == 1 }}
- name: Install just & nextest
uses: taiki-e/install-action@0c5db7f7f897c03b771660e91d065338615679f4 # v2.60.0
with:
Expand Down Expand Up @@ -119,6 +120,9 @@ jobs:
uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4
- name: Run setup
uses: ./.github/actions/setup
with:
additional-cache-key: ${{ matrix.package }}-${{ matrix.test_flags || 'default' }}-${{ matrix.lint_flags || 'default' }}
rust-cache-save: ${{ startsWith(matrix.partition || '1/1', '1/') }}
- name: Install just
uses: taiki-e/install-action@0c5db7f7f897c03b771660e91d065338615679f4 # v2.60.0
with:
Expand Down Expand Up @@ -174,6 +178,8 @@ jobs:
uses: ./.github/actions/setup
with:
additional-toolchain: ${{ env.NIGHTLY_VERSION }}
additional-cache-key: ${{ matrix.fuzz_dir }}
rust-cache-save: ${{ startsWith(matrix.partition || '1/1', '1/') }}
- name: Get Rust version
id: rust-version
run: echo "rust_version=$(rustc +${{ env.NIGHTLY_VERSION }} --version)" >> "$GITHUB_OUTPUT"
Expand Down Expand Up @@ -230,7 +236,7 @@ jobs:
with:
toolchain: ${{ env.NIGHTLY_VERSION }}
components: miri
additional-cache-key: part-${{ matrix.partition }}
rust-cache-save: ${{ matrix.partition == 1 }}
- name: Install miri
run: cargo miri setup
- name: Install just & nextest
Expand Down Expand Up @@ -285,7 +291,7 @@ jobs:
- name: Run setup
uses: ./.github/actions/setup
with:
additional-cache-key: ${{ matrix.name }}-part-${{ matrix.partition }}
rust-cache-save: ${{ matrix.partition == 1 }}
- name: Install just
uses: taiki-e/install-action@0c5db7f7f897c03b771660e91d065338615679f4 # v2.60.0
with:
Expand Down
Loading