Re-pin the service Dockerfiles to base-image digests during the release, not to the CalVer tag

[swinney]

Objective

Make the release workflow re-pin the service Dockerfiles to the digests of the base images it just pushed, instead of leaving them on the mutable CalVer tag, so a released tree is as reproducible as a development tree.

Context you need

Deferred from the review of PR #370 (fix/issue-335-pin-service-dockerfiles-to-digests, head 57269b6d). Codex raised the underlying conflict as P1 on tests/unit/test_base_image_preflight.py (comment id 3870377617) and offered two remedies. #370 took the narrower one — teach the guards that a release-retargeted tree is legitimate. This issue carries the other one, which is strictly better but larger.

Measured state on 57269b6d:

• The release workflow's "Point Dockerfiles to versioned base images" step (test-and-build-tag.yml:158) runs update_service_base_images.py --tag <release-tag> --switch-source ghcr --orig-tag all. That rewrites all 15 templates from FROM ghcr.io/fasrc/<base>@sha256:<64hex> to FROM ghcr.io/fasrc/<base>:v2026.08.0.
• The commit step at test-and-build-tag.yml:207 commits and pushes that tree to the dispatched ref.
• Verified by running the documented invocation against a copy of the templates: all 15 lose the digest and lose the # base-image-pin: ... (managed by update_service_base_images.py) annotation line above the FROM.
• update_service_base_images.py already supports digest pinning: --digest NAME=sha256:HEX, repeatable, with --bases {python,pytorch}. So the writing half exists; what is missing is resolving the just-pushed tag to its digest inside the workflow.
• tests/unit/test_base_image_preflight.py (as of Pin the 15 service Dockerfile templates to ghcr digests #370) classifies a reference as digest, release, or mutable via _pin_state(), and test_all_templates_share_one_pin_state requires the 15 to agree.

Why this is worth doing: a tag is a mutable pointer. ghcr.io/fasrc/a2rchi-python-base:v2026.08.0 can be repushed to point at different bytes, so a rebuild of a released tag is not guaranteed to reproduce the released image — which is the exact property #335 was opened to establish. Accepting the release tag in #370 was the honest description of current behaviour, not an endorsement of it.

Constraints

• Branch from origin/dev. Open the PR against fasrc/archi:dev — never upstream/dev.
• No Co-Authored-By or session trailers on commits.
• TDD: failing test first, then the minimum code, then the gate.
• bash scripts/gate.sh must exit 0 before every commit. Never --no-verify.
• Do not weaken test_no_template_carries_a_mutable_base_reference. The dev-<sha> tag must stay rejected.
• The release workflow pushes to the dispatched ref by name, so it must be dispatched with a branch, not a SHA. Do not change that; just do not make it worse.
• Changing a release workflow cannot be validated by unit tests alone. Say in the PR body how you exercised it (a dispatch against a throwaway branch, or a dry-run job).

Plan

1. In the release workflow, after build-images pushes and before the retarget step commits, resolve each pushed base image tag to its digest — docker buildx imagetools inspect ghcr.io/fasrc/<base>:<tag> --format '{{.Manifest.Digest}}', or the equivalent crane digest.
2. Replace the --tag ... --switch-source ghcr --orig-tag all invocation with the --digest <name>=sha256:<hex> form for both bases, so the committed tree carries digests.
3. Confirm the annotation line survives, or is rewritten, so update_service_base_images.py can still locate the pins on a released tree. It is dropped by the tag form today.
4. Update the two --verify steps (:168 and :307) so they verify the digest, not the tag. Verifying the tag against a digest-pinned tree would fail closed and block the release.
5. Once the released tree carries digests, remove the release branch from _pin_state() in tests/unit/test_base_image_preflight.py and delete the pytest.skip in test_base_references_are_pinned_to_the_expected_digests, so a tag-shaped reference is rejected again everywhere. Keep test_the_guards_accept_a_release_retargeted_tree only if the tag form remains reachable; delete it if it does not.

Commands

git fetch origin && git checkout -b fix/issue-<this>-release-digest-repin origin/dev
python scripts/dev/update_service_base_images.py --help
python -m pytest tests/unit/test_base_image_preflight.py -q
bash scripts/gate.sh                    # run bare; it refuses to run piped or redirected

To reproduce the current released state without touching the repo tree, copy scripts/dev/update_service_base_images.py and src/cli/templates/dockerfiles/ into a temp directory with the same relative layout and run the updater there — the script resolves its target from Path(__file__).resolve().parents[2]. _materialize_release_retargeted_tree in tests/unit/test_base_image_preflight.py already does exactly this; reuse it.

Acceptance criteria

• bash scripts/gate.sh exits 0.
• After the release workflow's retarget step, every one of the 15 templates carries @sha256: — asserted by a test that runs the real updater invocation the workflow uses.
• Both --verify steps pass against the digest-pinned tree.
• test_no_template_carries_a_mutable_base_reference still fails when a template is rewritten to dev-<sha>. Prove it in the PR body by pasting the failing run.
• The release state is removed from _pin_state(), or the PR explains in the body why it must stay.
• The PR body records how the workflow change was exercised, since unit tests cannot reach it.
• The PR targets fasrc/archi:dev and its body carries a Closes #<this> line (the body, not the title).

Start here

Read test-and-build-tag.yml lines 150-210 end to end, then run python scripts/dev/update_service_base_images.py --help and confirm the --digest form writes the annotation line. That single fact decides whether step 3 is a no-op or the bulk of the work.

[swinney]

Originating review thread: #370 (comment) (PR #370, Codex P1 on tests/unit/test_base_image_preflight.py).