Skip to content

Commit 73b8a47

Browse files
mauro-agentmauromorales
mauro-agent
and
mauromorales
authored
ci: gate releases on a vulnerability scan of the shipped bundle (#4300)
* ci: gate releases on a vulnerability scan of the shipped bundle Adds a scan of the binaries Kairos actually ships, for the kairos-init version this repo pins. The existing scanning cannot see these dependencies. kairos-init UPX-compresses the bundled binaries, and a UPX-packed Go binary no longer exposes its module metadata, so immucore, kairos-agent, provider-kairos and the rest are opaque to a scanner. release.yaml's grype and trivy runs are also report-only with grype_sarif_fail_build disabled, so a green release says nothing about CVEs. The job resolves ARG KAIROS_INIT from images/Dockerfile, checks kairos-init out at that tag, runs `make download` with SKIP_UPX=true so the binaries stay readable, and scans the result with osv-scanner. The scanned artifact is deliberately not byte-identical to the shipped one; it is composed from the same sources at the same pins, which is what matters for dependency scanning. Two scanner flags are load-bearing and are commented as such, because without either the scan finds nothing and reports success: --no-ignore kairos-init's .gitignore excludes pkg/bundled/binaries/, and osv-scanner honours .gitignore. --experimental-plugins artifact the default plugin set cannot read Go binaries at all. A separate step fails the build when zero packages were extracted, so a broken scan cannot pass as a clean one. osv-scanner.toml carries the ignore list. Anything with a published, takeable fix is meant to be bumped rather than ignored; entries are limited to what cannot currently be fixed, each with a reason and an expiry, and an expired entry fails the build. Expiry varies by category so the gate stays trustworthy: 90 days where no fix exists upstream, two weeks where the only fix is a Go prerelease, thirty days where clearing it needs a major upgrade. Runs on pull_request so master stays releasable, and exposes workflow_call so release.yaml can depend on it. Refs: #3985 Signed-off-by: Mauro Morales <contact@mauromorales.com> * ci: scope the scan triggers to releases and the files that decide the result Review feedback: this is a release gate, so it should not run on every pull request. Correct, and the unscoped trigger was actively harmful. The findings belong to the pinned dependency set, not to the change under review, so with advisories outstanding every open PR would show a red check for CVEs its author did not introduce and cannot fix -- including external contributors' PRs on a public repo. New advisories are published continuously, so a PR green yesterday would go red today for reasons unrelated to it. workflow_call is the gate; release.yaml depends on it. workflow_dispatch stays for manual runs. pull_request is kept but scoped to the three files that actually determine the outcome: the kairos-init pin in images/Dockerfile, the ignore list, and this workflow. That keeps a bad pin bump or a bad ignore entry caught where it is relevant, and means edits to the workflow itself are still exercised in CI, without touching any unrelated PR. Signed-off-by: Mauro Morales <contact@mauromorales.com> * ci: make the scan an actual release gate The previous commit exposed workflow_call but nothing called it, so a workflow named "Release vulnerability scan" never ran on a release. It gated nothing. release.yaml and release-arm.yaml now both start with a vulnerability-scan job, and every publishing job depends on it: core, standard-k3s and standard-k0s on amd64, core and standard on arm64. A tag push runs the scan first, and if the shipped bundle carries unignored advisories the factory jobs never run, so no artifacts and no GitHub release are produced. The scan resolves the kairos-init pin from images/Dockerfile, so it always describes the composition of the release being cut rather than whatever is current on master. One scan covers both architectures. It reads Go module metadata from the amd64 bundle, and the module set is the same across architectures because it comes from the same sources at the same pins. Signed-off-by: Mauro Morales <contact@mauromorales.com> * ci: distinguish advisories from findings in the scan report The first CI run reported "14 unignored advisories" while the scan had found 13. The scan was right and the message was wrong. One advisory can affect several module versions in the same bundle, because different binaries pin different versions -- GO-2026-5970 is present at both golang.org/x/text@0.37.0 and @0.38.0. The count was over rows, one per (advisory, module@version) pair, and every row was being called an advisory. Counts both now: advisories for the headline, findings for the per-module rows, both in the error and the step summary. The gate condition is unchanged. Signed-off-by: Mauro Morales <contact@mauromorales.com> * ci: ignore two advisories whose fixes break edgevpn connectivity Adds an upstream-regression category for the case the existing three did not cover: a fix is published, but taking it breaks the component. Both advisories are in the edgevpn binary, which ships in every Kairos release from mudler/edgevpn. Bisected against edgevpn's pkg/node suite, which exercises real peer-to-peer connectivity. Both make the "nodes can write to the ledger" spec time out at 240s where an unmodified checkout passes in 10-13 seconds. webtransport-go v0.11.1 pulls quic-go 0.59.1 -> 0.60.0 and that alone reproduces the failure; go-libp2p pins quic-go tightly, so this likely needs a go-libp2p bump first. x/text v0.39.0 passes alone but fails combined with the pion bumps, and x/text supplies IDNA which multiaddr uses for DNS peer addresses. The two advisories in the same scan that could be taken safely are in mudler/edgevpn#1075. These two were deliberately left out of it. Expiry is 60 days rather than the 30 used for major-upgrade-required: the work is in a project we do not control, so a shorter clock would redden the gate without giving anyone an action. Signed-off-by: Mauro Morales <contact@mauromorales.com> * ci: fix the false-green guard failing on clean scans, ignore two pending fixes Two changes, one a real bug found while verifying the provider-kairos v2.16.2 bump. The guard was inverted in effect. osv-scanner's default JSON only emits results for packages that HAVE findings, so a scan with nothing to report produces an empty results array -- identical, to the guard, to a scan that read no binaries at all. It would have failed every genuinely clean release, which is the opposite of what it exists for. Adding --all-packages makes the output list everything it read: the same bundle now reports 1425 packages and 0 vulnerabilities, so "read nothing" and "found nothing" are finally distinguishable. Also adds an upstream-fix-pending category and the two advisories it covers. Both are pion parsing DoS issues in the edgevpn binary. provider-kairos v2.16.2 cleared its copies; the edgevpn copies need mudler/edgevpn#1075 merged and an edgevpn release before EDGEVPN_VERSION can move. Thirty days, shorter than upstream-regression, because this one is expected to resolve. Verified against the real bundle at provider-kairos v2.16.2: guard passes on 1425 packages, enforce passes with zero unignored advisories. Signed-off-by: Mauro Morales <contact@mauromorales.com> --------- Signed-off-by: Mauro Morales <contact@mauromorales.com> Co-authored-by: Mauro Morales <contact@mauromorales.com>
1 parent be1a660 commit 73b8a47

4 files changed

Lines changed: 389 additions & 0 deletions

File tree

‎.github/workflows/release-arm.yaml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,19 @@ on:
66
- v*
77
permissions: read-all
88
jobs:
9+
# Release gate: refuse to publish artifacts containing known-vulnerable
10+
# dependencies. Scans the binaries this release will actually ship, for the
11+
# kairos-init version pinned in images/Dockerfile. See
12+
# .github/workflows/release-scan.yml and
13+
# https://github.com/kairos-io/kairos/issues/3985.
14+
vulnerability-scan:
15+
uses: ./.github/workflows/release-scan.yml
16+
917
core:
1018
name: ${{ matrix.image_name }}
1119
uses: kairos-io/kairos-factory-action/.github/workflows/reusable-factory.yaml@8952c3a5f4f6255cc63af88cb3637a9684970c8d # v1.1.3
20+
needs:
21+
- vulnerability-scan
1222
secrets:
1323
registry_username: ${{ secrets.QUAY_USERNAME }}
1424
registry_password: ${{ secrets.QUAY_PASSWORD }}
@@ -114,6 +124,7 @@ jobs:
114124
registry_username: ${{ secrets.QUAY_USERNAME }}
115125
registry_password: ${{ secrets.QUAY_PASSWORD }}
116126
needs:
127+
- vulnerability-scan
117128
- get-k3s-versions
118129
permissions:
119130
id-token: write # OIDC support

‎.github/workflows/release-scan.yml‎

Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
---
2+
name: 'Release vulnerability scan'
3+
4+
# Scans the binaries Kairos actually ships, for the kairos-init version this
5+
# repo pins. Refs https://github.com/kairos-io/kairos/issues/3985.
6+
#
7+
# Why this exists as a separate job rather than relying on the scanning already
8+
# in release.yaml: kairos-init UPX-compresses the bundled binaries, and a
9+
# UPX-packed Go binary no longer exposes its module metadata, so the
10+
# dependencies we ship are invisible to a scanner. This job rebuilds the bundle
11+
# with SKIP_UPX=true purely so it can be read.
12+
#
13+
# The scanned artifact is therefore NOT byte-identical to the shipped one. It is
14+
# composed from the same sources at the same pins, which is the property that
15+
# matters for dependency scanning.
16+
#
17+
18+
# Triggers are deliberately narrow. This is a release gate: release.yaml and
19+
# release-arm.yaml call it via workflow_call, and every publishing job depends
20+
# on it, so a release with unignored advisories never publishes. It does NOT run on every pull request: the
21+
# findings belong to the pinned dependency set, not to the change under review,
22+
# so an unrelated PR -- including an external contributor's -- would get a red
23+
# check for CVEs it did not introduce and cannot fix.
24+
#
25+
# The pull_request trigger is scoped to the files that actually determine the
26+
# result: the kairos-init pin, the ignore list, and this workflow itself.
27+
on:
28+
workflow_call:
29+
workflow_dispatch:
30+
pull_request:
31+
paths:
32+
- 'images/Dockerfile'
33+
- 'osv-scanner.toml'
34+
- '.github/workflows/release-scan.yml'
35+
36+
permissions:
37+
contents: read
38+
39+
jobs:
40+
scan-shipped-bundle:
41+
name: scan shipped bundle
42+
runs-on: ubuntu-latest
43+
steps:
44+
- name: Checkout kairos
45+
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
46+
47+
- name: Resolve the pinned kairos-init version
48+
id: pin
49+
run: |
50+
set -euo pipefail
51+
VERSION="$(grep -m1 '^ARG KAIROS_INIT=' images/Dockerfile | cut -d= -f2)"
52+
if [ -z "${VERSION}" ]; then
53+
echo "::error::could not read ARG KAIROS_INIT from images/Dockerfile"
54+
exit 1
55+
fi
56+
echo "version=${VERSION}" >> "${GITHUB_OUTPUT}"
57+
echo "Scanning the bundle composed by kairos-init ${VERSION}"
58+
59+
- name: Checkout kairos-init at that version
60+
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
61+
with:
62+
repository: kairos-io/kairos-init
63+
ref: ${{ steps.pin.outputs.version }}
64+
path: kairos-init
65+
66+
# SKIP_UPX is the whole point of this job -- see the header comment.
67+
- name: Download the shipped binaries, uncompressed
68+
working-directory: kairos-init
69+
run: SKIP_UPX=true make download
70+
71+
- name: Install osv-scanner
72+
env:
73+
OSV_SCANNER_VERSION: v2.5.0
74+
run: |
75+
set -euo pipefail
76+
curl -sSfL --retry 5 --retry-all-errors --retry-delay 2 \
77+
"https://github.com/google/osv-scanner/releases/download/${OSV_SCANNER_VERSION}/osv-scanner_linux_amd64" \
78+
-o /usr/local/bin/osv-scanner
79+
chmod +x /usr/local/bin/osv-scanner
80+
osv-scanner --version
81+
82+
# Two flags are load-bearing. Without either, the scan finds nothing and
83+
# reports success -- a false green, which is worse than no gate at all.
84+
#
85+
# --no-ignore kairos-init's .gitignore excludes
86+
# pkg/bundled/binaries/, and osv-scanner
87+
# honours .gitignore by default.
88+
# --experimental-plugins artifact the default plugin set
89+
# (lockfile, sbom, directory) cannot
90+
# read Go binaries at all.
91+
# --all-packages without it, a scan with no findings
92+
# emits an EMPTY results array, which
93+
# is indistinguishable from a scan that
94+
# read nothing -- and would trip the
95+
# false-green guard below on every
96+
# clean release.
97+
#
98+
# continue-on-error because osv-scanner exits non-zero when it finds
99+
# vulnerabilities; the reporting step below decides what that means.
100+
- name: Scan
101+
continue-on-error: true
102+
run: |
103+
set -uo pipefail
104+
osv-scanner scan source \
105+
--experimental-plugins artifact \
106+
--no-ignore \
107+
--recursive \
108+
--all-packages \
109+
--config osv-scanner.toml \
110+
--format json \
111+
--output-file osv.json \
112+
kairos-init/pkg/bundled/binaries
113+
114+
# A clean result only means something if the scanner actually read the
115+
# binaries. Zero extracted packages is a broken scan, not a pass.
116+
- name: Guard against a false green
117+
run: |
118+
set -euo pipefail
119+
if [ ! -s osv.json ]; then
120+
echo "::error::osv-scanner produced no output file"
121+
exit 1
122+
fi
123+
PACKAGES="$(jq '[.results[].packages[]] | length' osv.json)"
124+
echo "extracted packages: ${PACKAGES}"
125+
if [ "${PACKAGES}" -eq 0 ]; then
126+
echo "::error::scanner extracted 0 packages -- it did not read the shipped binaries. Treating as a failure, not as clean."
127+
exit 1
128+
fi
129+
130+
- name: Report and enforce
131+
run: |
132+
set -euo pipefail
133+
# One advisory can affect several module versions in the same bundle
134+
# (different binaries pin different versions), so a finding is an
135+
# (advisory, module@version) pair and the two counts differ. Report
136+
# both rather than calling every row an advisory.
137+
jq -r '
138+
[ .results[].packages[]
139+
| . as $pkg
140+
| .vulnerabilities[]?
141+
| "| `\(.id)` | `\($pkg.package.name)@\($pkg.package.version)` |"
142+
] | unique | .[]' osv.json > findings.md || true
143+
FINDINGS="$(wc -l < findings.md)"
144+
ADVISORIES="$(jq -r '[.results[].packages[].vulnerabilities[]?.id] | unique | length' osv.json)"
145+
{
146+
echo "## Vulnerability scan of the shipped bundle"
147+
echo
148+
echo "kairos-init \`${{ steps.pin.outputs.version }}\`, scanned uncompressed."
149+
echo
150+
if [ "${FINDINGS}" -eq 0 ]; then
151+
echo "No unignored advisories."
152+
else
153+
echo "**${ADVISORIES} advisories**, ${FINDINGS} findings across module versions."
154+
echo
155+
echo "| advisory | module |"
156+
echo "|---|---|"
157+
cat findings.md
158+
fi
159+
} >> "${GITHUB_STEP_SUMMARY}"
160+
if [ "${FINDINGS}" -gt 0 ]; then
161+
echo "::error::${ADVISORIES} unignored advisories (${FINDINGS} findings) in the shipped bundle. Bump the affected component, or add a dated entry to osv-scanner.toml with a reason."
162+
exit 1
163+
fi
164+
165+
- name: Upload scan output
166+
if: always()
167+
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
168+
with:
169+
name: osv-scan
170+
path: osv.json
171+
if-no-files-found: warn

‎.github/workflows/release.yaml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,19 @@ on:
66
- v*
77
permissions: read-all
88
jobs:
9+
# Release gate: refuse to publish artifacts containing known-vulnerable
10+
# dependencies. Scans the binaries this release will actually ship, for the
11+
# kairos-init version pinned in images/Dockerfile. See
12+
# .github/workflows/release-scan.yml and
13+
# https://github.com/kairos-io/kairos/issues/3985.
14+
vulnerability-scan:
15+
uses: ./.github/workflows/release-scan.yml
16+
917
core:
1018
name: ${{ matrix.image_name }}
1119
uses: kairos-io/kairos-factory-action/.github/workflows/reusable-factory.yaml@8952c3a5f4f6255cc63af88cb3637a9684970c8d # v1.1.3
20+
needs:
21+
- vulnerability-scan
1222
permissions:
1323
id-token: write # OIDC support
1424
contents: write
@@ -115,6 +125,7 @@ jobs:
115125
name: ${{ matrix.image_name }}
116126
uses: kairos-io/kairos-factory-action/.github/workflows/reusable-factory.yaml@8952c3a5f4f6255cc63af88cb3637a9684970c8d # v1.1.3
117127
needs:
128+
- vulnerability-scan
118129
- get-k3s-versions
119130
permissions:
120131
id-token: write # OIDC support
@@ -161,6 +172,7 @@ jobs:
161172
name: ${{ matrix.image_name }}
162173
uses: kairos-io/kairos-factory-action/.github/workflows/reusable-factory.yaml@8952c3a5f4f6255cc63af88cb3637a9684970c8d # v1.1.3
163174
needs:
175+
- vulnerability-scan
164176
- get-k0s-versions
165177
permissions:
166178
id-token: write # OIDC support

0 commit comments

Comments
 (0)