audit: :stable and :testing point at images that never passed e2e

[castrojo]

Summary

Users tracking ghcr.io/projectbluefin/bluefin:stable are running an image that never passed e2e. So are users on :testing. This is a follow-up audit to the publish_stream_tag stream-tag leak fixed in projectbluefin/actions#405.

This issue is an audit report — nothing in GHCR has been re-tagged or deleted, and no release behavior has been changed. It documents what happened so a human can decide the remediation.

Timeline of facts

1. post-testing-e2e has not passed since 2026-06-22

gh run list --repo projectbluefin/bluefin --workflow post-testing-e2e.yml -L 400:

Period	Result
Last success	2026-06-22 (runs 27922429853, 27922740150)
2026-06-23 → 2026-08-07	0 success, ~200 runs, all failure or skipped

The promote-to-testing job — the only legitimate writer of the :testing tag — has therefore not run since 2026-06-22.

2. …yet :testing kept moving anyway (the leak, now fixed)

reusable-build.yml's compute-push-tags published the bare stream tag on every build despite publish_stream_tag: "false", because just generate-build-tags emits testing as alias_tags[0]. Fixed in projectbluefin/actions#405 (43c34f6), live on @v1.

Registry proof that :testing was written by the build, not by a promotion — all three tags sit on one package version pushed at build time:

$ gh api "/orgs/projectbluefin/packages/container/bluefin/versions"
sha256:bf615b200faef  2026-08-04T16:42:15Z  testing-20260804,testing-44.20260804,testing

sha256:bf615b20… is the digest that failed e2e in run 30931237054 (smoke-a = failure, promote-to-testing = skipped).

3. :stable was promoted from that ungated :testing pointer, with no gate of its own

execute-release.yml / promote-bluefin-stable.yml both pass:

variants: '[{"image":"bluefin","source_tag":"testing","target_tag":"stable"}, ...]'

reusable-execute-release.yml's resolve job does skopeo inspect ghcr.io/projectbluefin/bluefin:testing — it trusts whatever :testing points at, by design. :testing is the release input.

The current bluefin:stable → sha256:383f5239… (label org.opencontainers.image.version=testing-44.20260720) was set by run 29716820591 on 2026-07-20T04:25 UTC:

Resolved bluefin:testing -> sha256:383f523982a84cd23b40a8d252b50d6c4a65b013b03bfb70e74e0708fd8e082f
Verifying ghcr.io/projectbluefin/bluefin@sha256:383f5239…
Promoting docker://…@sha256:383f5239… to docker://…:stable

That run's job list is: check-trigger, execute / execute, release-notes, post-release-variants. There is no release-gate job. The testsuite e2e gate was added to reusable-execute-release.yml in projectbluefin/actions#319, commit 9f3b74b, dated 2026-07-20T04:59 UTC — 34 minutes after this promotion ran. So :stable was set by the last ungated release, from an ungated :testing, during a window when e2e had already been failing for a month.

Net: two independent gates were both inert for the same digest.

• cosign verify did pass on that digest — but that proves provenance, not that the image works.

Is the release path safe now?

Yes, going forward — with one caveat.

• reusable-execute-release.yml now has release-gate (testsuite e2e against the exact resolved digest, fail-fast: true), and execute requires needs.release-gate.result == 'success' || 'skipped'.
• run_release_gate is a genuine type: boolean (default true), so it does not have the string-truthiness bug that caused the build-side leak. Both bluefin callers get the gate (execute-release.yml passes true; promote-bluefin-stable.yml omits it → default true).
• @v1 currently resolves to f8418f7, which includes the gate.

Caveat / latent fail-open: execute's condition treats release-gate == 'skipped' as a pass. That is correct for a deliberate run_release_gate: false, but it means any future change that causes the gate job to be skipped rather than fail silently re-opens the promotion path. Worth tightening to an explicit inputs.run_release_gate == false check rather than accepting skipped unconditionally.

:latest — stale, not leaked

bluefin:latest → sha256:d3aed20b… (testing-44.20260608.4, ~2026-06-08). No workflow in projectbluefin/bluefin or projectbluefin/actions writes bluefin:latest (grep target_tag finds only stable). It is an abandoned pointer roughly two months stale, not an active ungated promotion target. Anyone still tracking :latest is pinned to a June image.

User impact — stated plainly

• :testing users — running sha256:bf615b20…, an image whose e2e run explicitly failed. Ungated since roughly 2026-06-22.
• :stable users — running sha256:383f5239…, promoted 2026-07-20 from an ungated :testing by a release run that had no e2e gate. This image never passed e2e at any point in its lifecycle.
• :latest users — pinned to a stale June image; no active writer.
• Same class of leak affected projectbluefin/bluefin-lts (build-regular.yml, build-nvidia-aarch64.yml both pass publish_stream_tag: 'false'); also fixed by fix(ci): allow auto/promote-testing-to-main through base-branch check #405.

What needs a human decision

1. The real blocker is that e2e has been red for ~6.5 weeks. The tag leak hid it — :testing kept advancing, so the broken gate produced no visible symptom. Fixing the leak means :testing will now freeze until e2e is repaired. That is correct behavior, and it makes triaging the e2e failure urgent rather than optional.
2. Re-pointing :testing. The last digest that actually passed the gate is sha256:3655e13bd2e06a212b4284b57cbc22e95e879f4fee115bdcb9059d319408233c, promoted 2026-06-22 by run 27922740150. It is ~6.5 weeks old, so rolling :testing back to it is a real regression for users and probably not the right call — repairing e2e and promoting forward is likely better. Deliberately not done by any agent; user-visible, human decision.
3. :stable — same question, with higher stakes. There is no e2e-validated candidate newer than the 2026-06-22 digest.
4. Consider tightening the release-gate == 'skipped' fail-open noted above.

How this was verified

• skopeo inspect --no-tags docker://ghcr.io/projectbluefin/bluefin:{testing,stable,latest,gts}
• gh api /orgs/projectbluefin/packages/container/bluefin/versions (tag co-location proof)
• gh run view 30931237054 / 30926759229 / 29716820591 / 27922740150 --log
• gh run list --workflow post-testing-e2e.yml -L 400 (full success/failure census)
• git log -S "release-gate:" -- .github/workflows/reusable-execute-release.yml in projectbluefin/actions

Read-only throughout. No GHCR mutation, no release-workflow changes.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com