Skip to content

Commit 87b045b

Browse files
MattJacksonMattJackson
authored
ci: verify the container image is pullable before publishing a chart that points at it (#4)
release-on-upstream.yml self-heals off busbar's `releases/latest` on a daily cron and bumps `charts/busbar/Chart.yaml`'s appVersion the moment a newer release object appears. It never checked that the matching container image had been pushed. That is a real window, not a theoretical one. busbar's release.yml and docker.yml run in PARALLEL off the same tag push, so `releases/latest` can advance minutes before the image lands. values.yaml has `tag: ""`, which _helpers.tpl resolves to appVersion, so a chart published inside that window ImagePullBackOffs for every user who installs it. During the 1.5.3 release the window was open and nothing but timing kept a broken chart off ArtifactHub. The verifier reads the OCI Distribution API directly, which is the same bytes `docker pull` reads, rather than Docker Hub's tags index, which can lag hours behind a real push. It reads the image repository out of values.yaml instead of hardcoding it, so re-pointing the chart at another registry path cannot leave this check silently verifying the old one. On failure the chart simply stays where it is and the next cron retries, so this heals itself the moment the image appears. That is the same fail-closed-by-construction shape homebrew-busbar gets for free by downloading its tarballs under `set -euo pipefail`. Its self-test runs FIRST, before the real check, matching core's discipline of never trusting a gate's verdict before proving the gate still works: it asserts a real tag is accepted, a nonexistent tag is refused, and an unknown repository is refused. Verified both directions against the live registry.
1 parent b7e1a6c commit 87b045b

2 files changed

Lines changed: 123 additions & 0 deletions

File tree

‎.github/scripts/verify-image.sh‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
#!/usr/bin/env bash
2+
# Verify that a container image tag is actually PULLABLE before this repo publishes a chart that
3+
# points at it.
4+
#
5+
# WHY THIS EXISTS. release-on-upstream.yml self-heals off GetBusbar/busbar's `releases/latest` on a
6+
# daily cron. It used to bump `charts/busbar/Chart.yaml`'s appVersion the moment a newer release
7+
# object appeared, and never checked that the matching container image had been pushed. Those are
8+
# published by two DIFFERENT workflows (release.yml and docker.yml) that run in PARALLEL off the same
9+
# tag push, so there is a real window in which the release object exists and the image does not. If
10+
# the cron fires inside that window the chart publishes with `tag: ""` in values.yaml, which falls
11+
# back to appVersion, and every user who installs it gets ImagePullBackOff. During the 1.5.3 release
12+
# that window was open and the only reason it did not detonate was timing.
13+
#
14+
# The pattern being copied here is GetBusbar/homebrew-busbar's, which is safe BY CONSTRUCTION rather
15+
# than by check: it downloads every tarball under `set -euo pipefail`, so it dies before re-pinning
16+
# and stays fail-closed at the previous version. This script makes the chart fail-closed the same
17+
# way: no image, no bump, chart stays where it was, and the next cron retries.
18+
#
19+
# Reads the OCI Distribution API directly (the same bytes `docker pull` reads), never Docker Hub's
20+
# tags/search index, which can lag hours behind a real push.
21+
#
22+
# Usage: verify-image.sh <repository> <tag> [attempts] [sleep-seconds]
23+
# Example: verify-image.sh getbusbar/busbar 1.5.3
24+
#
25+
# Self-test (proves this script goes RED as well as GREEN, before its verdict is trusted anywhere):
26+
# verify-image.sh --selftest
27+
set -euo pipefail
28+
29+
ACCEPT='application/vnd.oci.image.index.v1+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.docker.distribution.manifest.v2+json'
30+
31+
# Prints the HTTP status of a HEAD-equivalent manifest fetch. Anonymous pull token, because that is
32+
# exactly the credential a chart user has.
33+
manifest_status() {
34+
local repo="$1" tag="$2" token
35+
token="$(curl -fsS --max-time 30 \
36+
"https://auth.docker.io/token?service=registry.docker.io&scope=repository:${repo}:pull" \
37+
| python3 -c 'import json,sys; print(json.load(sys.stdin)["token"])')" || return 1
38+
curl -sS --max-time 30 -o /dev/null -w '%{http_code}' \
39+
-H "Authorization: Bearer ${token}" -H "Accept: ${ACCEPT}" \
40+
"https://registry-1.docker.io/v2/${repo}/manifests/${tag}"
41+
}
42+
43+
verify() {
44+
local repo="$1" tag="$2" attempts="${3:-10}" nap="${4:-30}" i status
45+
for (( i = 1; i <= attempts; i++ )); do
46+
status="$(manifest_status "$repo" "$tag" || echo 000)"
47+
if [ "$status" = "200" ]; then
48+
echo "OK: ${repo}:${tag} is pullable from registry-1.docker.io (HTTP 200, attempt ${i}/${attempts})"
49+
return 0
50+
fi
51+
echo "not yet: ${repo}:${tag} -> HTTP ${status} (attempt ${i}/${attempts})"
52+
[ "$i" -lt "$attempts" ] && sleep "$nap"
53+
done
54+
echo "FAIL: ${repo}:${tag} is NOT pullable after ${attempts} attempt(s); last status HTTP ${status}." >&2
55+
echo "Refusing to publish a chart whose appVersion points at an image that does not exist:" >&2
56+
echo "every user who installed it would get ImagePullBackOff. The chart stays at its current" >&2
57+
echo "version and the next scheduled run will retry, so this heals itself once the image lands." >&2
58+
return 1
59+
}
60+
61+
if [ "${1:-}" = "--selftest" ]; then
62+
# A verifier nobody has watched go RED is not a verifier. Both directions, against the real
63+
# registry, before any caller trusts a verdict from this file.
64+
echo "== selftest: a tag that really exists must be accepted =="
65+
verify getbusbar/busbar latest 3 2 || { echo "SELFTEST FAILED: known-good tag was rejected" >&2; exit 1; }
66+
echo "== selftest: a tag that cannot exist must be refused (expect FAIL below) =="
67+
if verify getbusbar/busbar 0.0.0-does-not-exist 2 1; then
68+
echo "SELFTEST FAILED: a nonexistent tag was accepted. This script would wave through exactly" >&2
69+
echo "the broken-chart publish it exists to prevent." >&2
70+
exit 1
71+
fi
72+
echo "== selftest: an unknown repository must be refused (expect FAIL below) =="
73+
if verify getbusbar/no-such-image-at-all 1.0.0 1 1; then
74+
echo "SELFTEST FAILED: an unknown repository was accepted." >&2
75+
exit 1
76+
fi
77+
echo
78+
echo "SELFTEST PASSED: accepts a real tag, refuses a missing tag, refuses a missing repository."
79+
exit 0
80+
fi
81+
82+
if [ "$#" -lt 2 ]; then
83+
echo "usage: $0 <repository> <tag> [attempts] [sleep-seconds] | $0 --selftest" >&2
84+
exit 2
85+
fi
86+
verify "$@"

‎.github/workflows/release-on-upstream.yml‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,43 @@ jobs:
8989
echo "target=$target"
9090
} >> "$GITHUB_OUTPUT"
9191
92+
# FAIL-CLOSED ON A MISSING IMAGE. Everything above this step trusts busbar's RELEASE OBJECT.
93+
# The chart does not ship a release object, it ships an image reference: values.yaml has
94+
# `tag: ""`, which _helpers.tpl resolves to Chart.yaml's appVersion. So bumping appVersion to a
95+
# version whose container image has not been pushed publishes a chart that ImagePullBackOffs
96+
# for every user who installs it.
97+
#
98+
# That is a real window, not a theoretical one: busbar's release.yml and docker.yml run in
99+
# PARALLEL off the same tag push, so `releases/latest` can advance minutes before the image
100+
# lands, and this workflow's `19 7 * * *` cron does not care what time of day that is. During
101+
# the 1.5.3 release the window was open and nothing but timing kept a broken chart off
102+
# ArtifactHub.
103+
#
104+
# The verifier reads the OCI Distribution API (what `docker pull` actually reads), not Docker
105+
# Hub's tags index, which can lag hours behind a real push. Its self-test runs FIRST, the same
106+
# "prove the gate before you trust its verdict" discipline core uses on every lint: a verifier
107+
# that had rotted into always-passing would otherwise wave through exactly the broken publish
108+
# it exists to prevent.
109+
#
110+
# On failure the chart simply stays where it is and the next cron retries, so this heals
111+
# itself the moment the image appears. That is the same fail-closed-by-construction shape
112+
# GetBusbar/homebrew-busbar gets for free by downloading its tarballs under `set -euo pipefail`.
113+
- name: Verify the container image is pullable (self-test first)
114+
if: steps.resolve.outputs.act == 'yes'
115+
env:
116+
TARGET: ${{ steps.resolve.outputs.target }}
117+
run: |
118+
set -euo pipefail
119+
chmod +x .github/scripts/verify-image.sh
120+
.github/scripts/verify-image.sh --selftest
121+
# Read the repository from values.yaml rather than hardcoding it, so a chart that
122+
# re-points at another registry path cannot leave this check silently verifying the old
123+
# one. The tag is the appVersion we are about to write, which is what the chart will
124+
# resolve at install time.
125+
repo="$(python3 -c "import re,sys; print(re.search(r'^\s*repository:\s*(\S+)', open('charts/busbar/values.yaml').read(), re.M).group(1))")"
126+
echo "chart will reference ${repo}:${TARGET}"
127+
.github/scripts/verify-image.sh "$repo" "$TARGET" 10 30
128+
92129
- name: Bump chart appVersion + version to target
93130
if: steps.resolve.outputs.act == 'yes'
94131
env:

0 commit comments

Comments
 (0)