Skip to content

Commit a6ce714

Browse files
SebTardifSebTardif
authored
ci: cache tool binaries to improve CI speed and reliability (#186)
* ci: cache tool binaries to improve CI speed and reliability Add two reusable composite actions for tool caching: - install-go-tool: caches Go tool binaries compiled via go install - install-binary-tool: caches pre-built binaries downloaded via curl Apply caching to all CI tool installations across workflows: - Go tools: golangci-lint, gotestsum, govulncheck, benchstat, controller-gen, helm-docs, setup-envtest, ko (previously recompiled from source every run) - Binary tools: gitleaks, k3d, helm-unittest (previously re-downloaded every run) - FOSSA CLI: replaced fossas/fossa-action with cached direct download, pinned to v3.17.10 (fixes transient CDN failures like the one in PR #184) - pip packages: use actions/setup-python for yamllint and mkdocs-material Each tool gets its own cache entry keyed by name, OS, arch, and version. Version bumps automatically invalidate only the affected tool's cache. Closes #185 Signed-off-by: Sebastien Tardif <sebtardif@ncf.ca> * fix: create helm plugins directory before symlinking helm-unittest The Link helm-unittest plugin step failed because the parent directory $(helm env HELM_PLUGINS) did not exist on a fresh runner. Add mkdir -p to ensure the directory tree exists before ln -s. Signed-off-by: Sebastien Tardif <sebtardif@ncf.ca> --------- Signed-off-by: Sebastien Tardif <sebtardif@ncf.ca>
1 parent 78faa77 commit a6ce714

7 files changed

Lines changed: 233 additions & 124 deletions

File tree

‎.github/actions/install-binary-tool/action.yaml‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
name: Install Binary Tool (cached)
2+
description: |
3+
Download a pre-built binary and cache it across runs.
4+
On cache hit the binary is restored without re-downloading.
5+
On cache miss the install-command runs and the result is saved.
6+
7+
inputs:
8+
name:
9+
description: Tool name used as cache key prefix (e.g. gitleaks, k3d)
10+
required: true
11+
version:
12+
description: Version string for the cache key (e.g. 8.30.1)
13+
required: true
14+
install-command:
15+
description: Shell command that installs the binary into $TOOL_DIR
16+
required: true
17+
18+
runs:
19+
using: composite
20+
steps:
21+
- name: Restore ${{ inputs.name }} from cache
22+
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
23+
id: tool-cache
24+
with:
25+
path: ${{ runner.temp }}/bin-tool-cache/${{ inputs.name }}
26+
key: bin-tool-${{ inputs.name }}-${{ runner.os }}-${{ runner.arch }}-${{ inputs.version }}
27+
28+
- name: Install ${{ inputs.name }}
29+
if: steps.tool-cache.outputs.cache-hit != 'true'
30+
shell: bash -Eeuo pipefail {0}
31+
env:
32+
TOOL_DIR: ${{ runner.temp }}/bin-tool-cache/${{ inputs.name }}
33+
run: |
34+
mkdir -p "$TOOL_DIR"
35+
${{ inputs.install-command }}
36+
37+
- name: Add ${{ inputs.name }} to PATH
38+
shell: bash -Eeuo pipefail {0}
39+
run: echo "${RUNNER_TEMP}/bin-tool-cache/${{ inputs.name }}" >> "$GITHUB_PATH"

‎.github/actions/install-go-tool/action.yaml‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
name: Install Go Tool (cached)
2+
description: |
3+
Install a Go tool binary and cache it across runs.
4+
On cache hit the binary is restored from cache without recompilation.
5+
On cache miss the tool is compiled via 'go install' and saved for future runs.
6+
7+
inputs:
8+
name:
9+
description: Binary name produced by go install (e.g. gotestsum, govulncheck)
10+
required: true
11+
version:
12+
description: Version tag (e.g. v1.13.0)
13+
required: true
14+
package:
15+
description: Go module path (e.g. gotest.tools/gotestsum)
16+
required: true
17+
18+
runs:
19+
using: composite
20+
steps:
21+
- name: Restore ${{ inputs.name }} from cache
22+
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
23+
id: tool-cache
24+
with:
25+
path: ${{ runner.temp }}/go-tool-cache/${{ inputs.name }}
26+
key: go-tool-${{ inputs.name }}-${{ runner.os }}-${{ runner.arch }}-${{ inputs.version }}
27+
28+
- name: Install ${{ inputs.name }}
29+
if: steps.tool-cache.outputs.cache-hit != 'true'
30+
shell: bash -Eeuo pipefail {0}
31+
run: |
32+
tool_dir="${RUNNER_TEMP}/go-tool-cache/${{ inputs.name }}"
33+
mkdir -p "$tool_dir"
34+
GOBIN="$tool_dir" go install "${{ inputs.package }}@${{ inputs.version }}"
35+
36+
- name: Add ${{ inputs.name }} to PATH
37+
shell: bash -Eeuo pipefail {0}
38+
run: echo "${RUNNER_TEMP}/go-tool-cache/${{ inputs.name }}" >> "$GITHUB_PATH"

‎.github/actions/setup-e2e-cluster/action.yaml‎

Lines changed: 12 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -61,15 +61,12 @@ runs:
6161
/tmp/stress-ng.tar
6262
key: e2e-images-${{ runner.os }}-cm${{ inputs.cert-manager-version }}-prom${{ inputs.prometheus-chart-version }}-stress0.20.01
6363

64-
- name: Install k3d
65-
shell: bash -Eeuo pipefail -x {0}
66-
run: |
67-
mkdir -p "$HOME/.local/bin"
68-
if command -v k3d &>/dev/null; then
69-
echo "k3d already installed: $(k3d version)"
70-
else
71-
curl -s https://raw.githubusercontent.com/k3d-io/k3d/${{ inputs.k3d-version }}/install.sh | K3D_INSTALL_DIR="$HOME/.local/bin" USE_SUDO=false bash
72-
fi
64+
- uses: ./.github/actions/install-binary-tool
65+
with:
66+
name: k3d
67+
version: ${{ inputs.k3d-version }}
68+
install-command: |
69+
curl -s https://raw.githubusercontent.com/k3d-io/k3d/${{ inputs.k3d-version }}/install.sh | K3D_INSTALL_DIR="$TOOL_DIR" USE_SUDO=false bash
7370
7471
- name: Prepare isolated kubeconfig
7572
shell: bash -Eeuo pipefail -x {0}
@@ -78,7 +75,6 @@ runs:
7875
- name: Cleanup stale k3d clusters and Docker resources
7976
shell: bash -Eeuo pipefail -x {0}
8077
run: |
81-
export PATH="$HOME/.local/bin:$PATH"
8278
for cluster in $(k3d cluster list -o json 2>/dev/null | jq -r '.[].name // empty'); do
8379
started=$(docker inspect --format '{{.State.StartedAt}}' "k3d-${cluster}-server-0" 2>/dev/null) || continue
8480
started_epoch=$(date -d "$started" +%s 2>/dev/null) || continue
@@ -109,7 +105,6 @@ runs:
109105
- name: Create k3d cluster
110106
shell: bash -Eeuo pipefail -x {0}
111107
run: |
112-
export PATH="$HOME/.local/bin:$PATH"
113108
for attempt in 1 2 3; do
114109
if k3d cluster create "${{ inputs.cluster-name }}" \
115110
--image ${{ inputs.k3s-image }} \
@@ -137,7 +132,6 @@ runs:
137132
- name: Load cert-manager images into cluster
138133
shell: bash -Eeuo pipefail -x {0}
139134
run: |
140-
export PATH="$HOME/.local/bin:$PATH"
141135
k3d image import \
142136
/tmp/cert-manager-controller.tar \
143137
/tmp/cert-manager-webhook.tar \
@@ -155,7 +149,6 @@ runs:
155149
- name: Load Prometheus and test images into cluster
156150
shell: bash -Eeuo pipefail -x {0}
157151
run: |
158-
export PATH="$HOME/.local/bin:$PATH"
159152
k3d image import /tmp/prometheus.tar /tmp/stress-ng.tar -c "${{ inputs.cluster-name }}"
160153
161154
- name: Install Prometheus
@@ -192,11 +185,15 @@ runs:
192185
sleep 5
193186
done
194187
188+
- uses: ./.github/actions/install-go-tool
189+
with:
190+
name: ko
191+
version: v0.18.0
192+
package: github.com/google/ko
193+
195194
- name: Build and load operator image
196195
shell: bash -Eeuo pipefail -x {0}
197196
run: |
198-
export PATH="$HOME/.local/bin:$PATH"
199-
go install github.com/google/ko@v0.18.0
200197
VERSION=e2e COMMIT=$(git rev-parse --short HEAD) DATE=$(date -u +%Y-%m-%dT%H:%M:%SZ) \
201198
KO_DOCKER_REPO=attune ko build ./cmd/manager/ \
202199
--bare --tags=e2e --platform=linux/$(go env GOARCH) \

‎.github/workflows/ci.yaml‎

Lines changed: 89 additions & 67 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ env:
2525
K3S_IMAGE: "rancher/k3s:v1.35.4-k3s1"
2626
ENVTEST_K8S_VERSION: "1.35.0"
2727
HELM_UNITTEST_VERSION: "v0.7.2"
28+
BENCHSTAT_VERSION: "v0.0.0-20260512194132-3cf34090a3db"
2829
CERT_MANAGER_VERSION: "v1.17.2"
2930
PROMETHEUS_IMAGE: "quay.io/prometheus/prometheus:v3.4.1"
3031
PROMETHEUS_CHART_VERSION: "27.22.0"
@@ -104,17 +105,15 @@ jobs:
104105
go-version: ${{ env.GO_VERSION }}
105106
cache: true
106107

108+
- uses: ./.github/actions/install-go-tool
109+
with:
110+
name: golangci-lint
111+
version: ${{ env.GOLANGCI_LINT_VERSION }}
112+
package: github.com/golangci/golangci-lint/v2/cmd/golangci-lint
113+
107114
- name: golangci-lint
108115
shell: bash -Eeuo pipefail {0}
109-
run: |
110-
lint_root=$(mktemp -d "${RUNNER_TEMP:-/tmp}/golangci-lint-${GITHUB_RUN_ID:-local}-${GITHUB_JOB:-lint}-XXXXXX")
111-
trap 'rm -rf "$lint_root"' EXIT
112-
lint_bindir="$lint_root/bin"
113-
lint_cachedir="$lint_root/cache"
114-
mkdir -p "$lint_bindir" "$lint_cachedir"
115-
GOBIN="$lint_bindir" make golangci-lint LOCALBIN="$lint_bindir"
116-
GOLANGCI_LINT_CACHE="$lint_cachedir" \
117-
"$lint_bindir/golangci-lint-${{ env.GOLANGCI_LINT_VERSION }}" run --timeout 5m --allow-serial-runners
116+
run: golangci-lint run --timeout 5m --allow-serial-runners
118117

119118
- name: Check go mod tidy
120119
shell: bash -Eeuo pipefail {0}
@@ -165,11 +164,14 @@ jobs:
165164
go-version: ${{ env.GO_VERSION }}
166165
cache: true
167166

167+
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
168+
with:
169+
python-version: "3.x"
170+
cache: pip
171+
168172
- name: Install MkDocs
169173
shell: bash -Eeuo pipefail {0}
170-
run: |
171-
python3 -m pip install --user --break-system-packages mkdocs-material==9.7.6
172-
echo "$(python3 -m site --user-base)/bin" >> "$GITHUB_PATH"
174+
run: pip install mkdocs-material==9.7.6
173175

174176
- name: Build docs site
175177
shell: bash -Eeuo pipefail {0}
@@ -219,13 +221,17 @@ jobs:
219221
egress-policy: audit
220222
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
221223

224+
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
225+
with:
226+
python-version: "3.x"
227+
228+
- name: Install yamllint
229+
shell: bash -Eeuo pipefail {0}
230+
run: pip install yamllint
231+
222232
- name: YAML lint
223233
shell: bash -Eeuo pipefail {0}
224234
run: |
225-
if ! command -v yamllint &>/dev/null; then
226-
python3 -m pip install --user --break-system-packages yamllint 2>/dev/null
227-
export PATH="$(python3 -m site --user-base)/bin:$PATH"
228-
fi
229235
yamllint_config="${RUNNER_TEMP:-/tmp}/yamllint-${GITHUB_RUN_ID}-${GITHUB_JOB}.yaml"
230236
cat > "$yamllint_config" <<'EOF'
231237
extends: default
@@ -277,13 +283,11 @@ jobs:
277283
go-version: ${{ env.GO_VERSION }}
278284
cache: true
279285

280-
- name: Install gotestsum
281-
shell: bash -Eeuo pipefail {0}
282-
run: |
283-
export GOBIN="${RUNNER_TEMP}/gobin-${GITHUB_RUN_ID}-${GITHUB_JOB}"
284-
mkdir -p "$GOBIN"
285-
go install gotest.tools/gotestsum@${{ env.GOTESTSUM_VERSION }}
286-
echo "${GOBIN}" >> "$GITHUB_PATH"
286+
- uses: ./.github/actions/install-go-tool
287+
with:
288+
name: gotestsum
289+
version: ${{ env.GOTESTSUM_VERSION }}
290+
package: gotest.tools/gotestsum
287291

288292
- name: Run unit tests
289293
shell: bash -Eeuo pipefail -x {0}
@@ -365,9 +369,11 @@ jobs:
365369
go-version: ${{ env.GO_VERSION }}
366370
cache: true
367371

368-
- name: Install benchstat
369-
shell: bash -Eeuo pipefail {0}
370-
run: go install golang.org/x/perf/cmd/benchstat@latest
372+
- uses: ./.github/actions/install-go-tool
373+
with:
374+
name: benchstat
375+
version: v0.0.0-20260512194132-3cf34090a3db
376+
package: golang.org/x/perf/cmd/benchstat
371377

372378
- name: Restore baseline benchmarks
373379
uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
@@ -428,22 +434,21 @@ jobs:
428434
go-version: ${{ env.GO_VERSION }}
429435
cache: true
430436

431-
- name: Install gotestsum
432-
shell: bash -Eeuo pipefail {0}
433-
run: |
434-
export GOBIN="${RUNNER_TEMP}/gobin-${GITHUB_RUN_ID}-${GITHUB_JOB}"
435-
mkdir -p "$GOBIN"
436-
go install gotest.tools/gotestsum@${{ env.GOTESTSUM_VERSION }}
437-
echo "${GOBIN}" >> "$GITHUB_PATH"
437+
- uses: ./.github/actions/install-go-tool
438+
with:
439+
name: gotestsum
440+
version: ${{ env.GOTESTSUM_VERSION }}
441+
package: gotest.tools/gotestsum
442+
443+
- uses: ./.github/actions/install-go-tool
444+
with:
445+
name: setup-envtest
446+
version: v0.24.1
447+
package: sigs.k8s.io/controller-runtime/tools/setup-envtest
438448

439-
- name: Setup envtest
449+
- name: Setup envtest assets
440450
shell: bash -Eeuo pipefail -x {0}
441-
run: |
442-
export GOBIN="${RUNNER_TEMP}/gobin-${GITHUB_RUN_ID}-${GITHUB_JOB}"
443-
mkdir -p "$GOBIN"
444-
go install sigs.k8s.io/controller-runtime/tools/setup-envtest@release-0.24
445-
echo "${GOBIN}" >> "$GITHUB_PATH"
446-
echo "KUBEBUILDER_ASSETS=$(setup-envtest use ${{ env.ENVTEST_K8S_VERSION }} -p path)" >> "$GITHUB_ENV"
451+
run: echo "KUBEBUILDER_ASSETS=$(setup-envtest use ${{ env.ENVTEST_K8S_VERSION }} -p path)" >> "$GITHUB_ENV"
447452

448453
- name: Run integration tests
449454
shell: bash -Eeuo pipefail -x {0}
@@ -543,7 +548,6 @@ jobs:
543548
if: always()
544549
shell: bash -Eeuo pipefail {0}
545550
run: |
546-
export PATH="$HOME/.local/bin:$PATH"
547551
k3d cluster delete "$K3D_CLUSTER_NAME" 2>/dev/null || true
548552
rm -f "$KUBECONFIG"
549553
@@ -564,11 +568,15 @@ jobs:
564568
go-version: ${{ env.GO_VERSION }}
565569
cache: true
566570

571+
- uses: ./.github/actions/install-go-tool
572+
with:
573+
name: govulncheck
574+
version: v1.3.0
575+
package: golang.org/x/vuln/cmd/govulncheck
576+
567577
- name: Run govulncheck
568578
shell: bash -Eeuo pipefail -x {0}
569-
run: |
570-
go install golang.org/x/vuln/cmd/govulncheck@v1.3.0
571-
govulncheck ./...
579+
run: govulncheck ./...
572580

573581
crd-freshness:
574582
name: CRD Freshness Check
@@ -629,40 +637,54 @@ jobs:
629637
--api-versions cert-manager.io/v1 > /dev/null
630638
done
631639
632-
- name: Run chart unit tests
633-
shell: bash -Eeuo pipefail -x {0}
640+
- uses: ./.github/actions/install-binary-tool
641+
with:
642+
name: helm-unittest
643+
version: ${{ env.HELM_UNITTEST_VERSION }}
644+
install-command: |
645+
# Remove stale helm-unittest plugins that conflict on self-hosted runners
646+
rm -rf "$(helm env HELM_PLUGINS)/helm-unittest.git" 2>/dev/null || true
647+
ASSET_VERSION="${{ env.HELM_UNITTEST_VERSION }}"
648+
ASSET_VERSION="${ASSET_VERSION#v}"
649+
case "$(uname -s)" in
650+
Linux) HU_OS=linux ;;
651+
Darwin) HU_OS=macos ;;
652+
*) echo "unsupported OS: $(uname -s)" >&2; exit 1 ;;
653+
esac
654+
case "$(uname -m)" in
655+
x86_64) HU_ARCH=amd64 ;;
656+
aarch64|arm64) HU_ARCH=arm64 ;;
657+
*) echo "unsupported arch: $(uname -m)" >&2; exit 1 ;;
658+
esac
659+
curl -fsSL "https://github.com/helm-unittest/helm-unittest/releases/download/${{ env.HELM_UNITTEST_VERSION }}/helm-unittest-${HU_OS}-${HU_ARCH}-${ASSET_VERSION}.tgz" \
660+
| tar xz -C "$TOOL_DIR"
661+
662+
- name: Link helm-unittest plugin
663+
shell: bash -Eeuo pipefail {0}
634664
run: |
635-
# Remove stale helm-unittest plugins that conflict on self-hosted runners
636-
rm -rf "$(helm env HELM_PLUGINS)/helm-unittest.git" 2>/dev/null || true
637665
PLUGINS_DIR="$(helm env HELM_PLUGINS)/unittest"
638-
mkdir -p "$PLUGINS_DIR"
639-
HELM_UNITTEST_ASSET_VERSION="${HELM_UNITTEST_VERSION#v}"
640-
case "$(uname -s)" in
641-
Linux) HU_OS=linux ;;
642-
Darwin) HU_OS=macos ;;
643-
*) echo "unsupported OS: $(uname -s)" >&2; exit 1 ;;
644-
esac
645-
case "$(uname -m)" in
646-
x86_64) HU_ARCH=amd64 ;;
647-
aarch64|arm64) HU_ARCH=arm64 ;;
648-
*) echo "unsupported arch: $(uname -m)" >&2; exit 1 ;;
649-
esac
650-
curl -fsSL "https://github.com/helm-unittest/helm-unittest/releases/download/${HELM_UNITTEST_VERSION}/helm-unittest-${HU_OS}-${HU_ARCH}-${HELM_UNITTEST_ASSET_VERSION}.tgz" \
651-
| tar xz -C "$PLUGINS_DIR"
652-
helm unittest charts/attune
666+
mkdir -p "$(dirname "$PLUGINS_DIR")"
667+
rm -rf "$PLUGINS_DIR"
668+
ln -s "${RUNNER_TEMP}/bin-tool-cache/helm-unittest" "$PLUGINS_DIR"
669+
670+
- name: Run chart unit tests
671+
shell: bash -Eeuo pipefail -x {0}
672+
run: helm unittest charts/attune
653673

654674
- uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6
655675
with:
656676
go-version: ${{ env.GO_VERSION }}
657677
cache: true
658678

679+
- uses: ./.github/actions/install-go-tool
680+
with:
681+
name: helm-docs
682+
version: v1.14.2
683+
package: github.com/norwoodj/helm-docs/cmd/helm-docs
684+
659685
- name: Helm docs freshness
660686
shell: bash -Eeuo pipefail {0}
661687
run: |
662-
export GOBIN="${RUNNER_TEMP}/gobin-${GITHUB_RUN_ID}-${GITHUB_JOB}"
663-
mkdir -p "$GOBIN"
664-
go install github.com/norwoodj/helm-docs/cmd/helm-docs@v1.14.2
665-
export PATH="${GOBIN}:${PATH}"
666688
helm-docs --chart-search-root charts/
667689
if ! git diff --quiet --exit-code charts/attune/README.md; then
668690
echo "::error::Helm README is stale. Run 'make helm-docs-gen' and commit."

0 commit comments

Comments
 (0)