fix(devcontainer-base): validate base updates before publishing (#825)

## Summary

- revert the devcontainer base from Fedora 45 to Fedora 44
- make every regular CI job build and load the checked-out devcontainer
base locally
- preserve the existing `build`, `check-format`, `lint`, and
kustomization-test check contexts
- preserve devcontainer registry cache export through Docker's
containerd image store
- validate base-image updates through the full downstream pipeline
before publishing
- publish the exact tested candidate as `latest` only after validation
succeeds
- keep pull-request builds non-publishing and serialize base-image
builds for the same ref

## Root cause

The Fedora 45 update moved the devcontainer to Python 3.15. `aiohttp`
did not provide a CPython 3.15 wheel, so the devcontainer build fell
back to a source build and failed because no compiler was installed.

The base-image workflow previously published `latest` as soon as the
base itself built. Its pull-request run did not exercise the downstream
devcontainer, so the incompatible image reached `latest` despite green
PR checks.

## CI behavior

Regular CI assigns the checked-out base a commit-specific tag, builds
and loads it on each isolated runner, and passes that tag into the
devcontainer build. The required checks therefore validate repository
source rather than depending on the already-published `latest`.

The regular `build` job enables Docker's containerd image store. That
keeps the default daemon-backed builder able to consume the locally
loaded candidate while also exporting the shared devcontainer registry
cache.

The base-image workflow separately builds one candidate, runs
`check-format`, `lint`, and the complete kustomization test suite
against it, then retags and pushes that exact local image on main and
scheduled runs. Pull-request runs never publish.

## Validation

- formatted `.github/workflows/checks.yml` with the repository's
Prettier version
- parsed the workflow with `yq`
- passed `git diff --check`
- verified all four job definitions build a local base and still invoke
`devcontainers/ci`
- verified the [regular build
log](https://github.com/marinatedconcrete/config/actions/runs/30874694197/job/91883694229)
contains both the commit-specific `DEVCONTAINER_BASE_TAG` and
`--cache-to ghcr.io/marinatedconcrete/config-devcontainer`
- [all nine regular
checks](https://github.com/marinatedconcrete/config/actions/runs/30874694197)
passed: build, formatting, lint, and all six kustomization tests
- [candidate-image
validation](https://github.com/marinatedconcrete/config/actions/runs/30874696854)
passed the full downstream pipeline; the PR publish step was skipped

Author: robarnold

.devcontainer/Containerfile

@@ -1,3 +1,5 @@
+ARG DEVCONTAINER_BASE_TAG=latest
+
 FROM ghcr.io/hadolint/hadolint:v2.14.0@sha256:27086352fd5e1907ea2b934eb1023f217c5ae087992eb59fde121dce9c9ff21e AS hadolint
 FROM registry.k8s.io/kustomize/kustomize:v5.8.1@sha256:899fcd3bc898160e62bcaf82932b0cb29ba38d16272353db2e7acbba82129429 AS kustomize
 
@@ -6,7 +8,7 @@ COPY .devcontainer/build_files /
 COPY ansible/requirements.txt /requirements.txt
 
 # hadolint ignore=DL3007
-FROM ghcr.io/marinatedconcrete/devcontainer-base:latest
+FROM ghcr.io/marinatedconcrete/devcontainer-base:${DEVCONTAINER_BASE_TAG}
 
 USER root
 

.devcontainer/devcontainer.json

@@ -1,6 +1,9 @@
 {
   "name": "config",
   "build": {
+    "args": {
+      "DEVCONTAINER_BASE_TAG": "${localEnv:DEVCONTAINER_BASE_TAG:latest}"
+    },
     "cacheFrom": "ghcr.io/marinatedconcrete/config-devcontainer",
     "context": "..",
     "dockerfile": "Containerfile"

.github/workflows/checks.yml

@@ -3,6 +3,9 @@ name: Run checks
 on:
   push:
 
+env:
+  DEVCONTAINER_BASE_TAG: candidate-${{ github.sha }}
+
 jobs:
   build:
     permissions:
@@ -12,16 +15,29 @@ jobs:
     steps:
       - name: Checkout Repo
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+      - name: Set up Docker
+        uses: docker/setup-docker-action@77e84dbf09b47d1e29270283c22f16145aa85ca1 # v5
+        with:
+          daemon-config: |
+            {
+              "features": {
+                "containerd-snapshotter": true
+              }
+            }
       - name: Login to GitHub Container Registry
         uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0 # v4
         with:
           registry: ghcr.io
           username: ${{ github.repository_owner }}
           password: ${{ secrets.GITHUB_TOKEN }}
-      - name: Set up Buildx
-        uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4
+      - name: Build Candidate Base Image
+        uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
         with:
-          driver: docker-container
+          context: images/devcontainer-base
+          file: images/devcontainer-base/Containerfile
+          load: true
+          push: false
+          tags: ghcr.io/${{ github.repository_owner }}/devcontainer-base:${{ env.DEVCONTAINER_BASE_TAG }}
       - name: Build devcontainer image
         uses: devcontainers/ci@513af61f4de4f75d37e4438f184ba4358f0fc1ca # v0.3.1900000450
         with:
@@ -38,6 +54,14 @@ jobs:
     steps:
       - name: Checkout Repo
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+      - name: Build Candidate Base Image
+        uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
+        with:
+          context: images/devcontainer-base
+          file: images/devcontainer-base/Containerfile
+          load: true
+          push: false
+          tags: ghcr.io/${{ github.repository_owner }}/devcontainer-base:${{ env.DEVCONTAINER_BASE_TAG }}
       - name: Check formatting
         uses: devcontainers/ci@513af61f4de4f75d37e4438f184ba4358f0fc1ca # v0.3.1900000450
         with:
@@ -53,6 +77,14 @@ jobs:
     steps:
       - name: Checkout Repo
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+      - name: Build Candidate Base Image
+        uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
+        with:
+          context: images/devcontainer-base
+          file: images/devcontainer-base/Containerfile
+          load: true
+          push: false
+          tags: ghcr.io/${{ github.repository_owner }}/devcontainer-base:${{ env.DEVCONTAINER_BASE_TAG }}
       - name: Validate configs
         uses: devcontainers/ci@513af61f4de4f75d37e4438f184ba4358f0fc1ca # v0.3.1900000450
         with:
@@ -77,6 +109,14 @@ jobs:
     steps:
       - name: Checkout Repo
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+      - name: Build Candidate Base Image
+        uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
+        with:
+          context: images/devcontainer-base
+          file: images/devcontainer-base/Containerfile
+          load: true
+          push: false
+          tags: ghcr.io/${{ github.repository_owner }}/devcontainer-base:${{ env.DEVCONTAINER_BASE_TAG }}
       - name: Validate configs
         uses: devcontainers/ci@513af61f4de4f75d37e4438f184ba4358f0fc1ca # v0.3.1900000450
         with:

.github/workflows/image-devcontainer-base.yml

@@ -15,14 +15,20 @@ on: # yamllint disable-line rule:truthy
       - .github/workflows/image-devcontainer-base.yml
       - images/devcontainer-base/**
   schedule:
-      - cron: '0 0 * * 0'  # Triggers every Sunday at midnight UTC
+    - cron: "0 0 * * 0" # Triggers every Sunday at midnight UTC
+
+concurrency:
+  cancel-in-progress: true
+  group: devcontainer-base-${{ github.ref }}
 
 jobs:
   build-container:
     runs-on: ubuntu-latest
     permissions:
       contents: read
       packages: write
+    env:
+      DEVCONTAINER_BASE_TAG: candidate-${{ github.sha }}
 
     steps:
       - name: Checkout Repo
@@ -33,15 +39,27 @@ jobs:
           registry: ghcr.io
           username: ${{ github.actor }}
           password: ${{ secrets.GITHUB_TOKEN }}
-      - name: Generate Metadata
-        uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6.2.0
-        id: metadata
-        with:
-          images: ghcr.io/${{ github.repository_owner }}/devcontainer-base
-      - name: Build Image
+      - name: Build Candidate Image
         uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
         with:
           context: images/devcontainer-base
           file: images/devcontainer-base/Containerfile
-          push: ${{ github.event_name != 'pull_request' }}
-          tags: ghcr.io/${{ github.repository_owner }}/devcontainer-base:latest
+          load: true
+          push: false
+          tags: ghcr.io/${{ github.repository_owner }}/devcontainer-base:${{ env.DEVCONTAINER_BASE_TAG }}
+      - name: Validate Full CI Pipeline
+        uses: devcontainers/ci@513af61f4de4f75d37e4438f184ba4358f0fc1ca # v0.3.1900000450
+        with:
+          cacheFrom: ghcr.io/marinatedconcrete/config-devcontainer
+          env: |
+            CI=1
+          imageName: config-devcontainer-candidate
+          push: never
+          runCmd: just check-format && just lint && just test
+      - name: Push Tested Image
+        if: github.event_name != 'pull_request'
+        run: |
+          docker tag \
+            "ghcr.io/${{ github.repository_owner }}/devcontainer-base:${DEVCONTAINER_BASE_TAG}" \
+            ghcr.io/${{ github.repository_owner }}/devcontainer-base:latest
+          docker push ghcr.io/${{ github.repository_owner }}/devcontainer-base:latest

images/devcontainer-base/Containerfile

@@ -1,7 +1,7 @@
 FROM scratch AS ctx
 COPY build_files /
 
-FROM quay.io/fedora/fedora-minimal:45
+FROM quay.io/fedora/fedora-minimal:44
 
 ARG USERNAME=vscode
 ARG USERID=1000