Skip to content

fix(devcontainer-base): validate base updates before publishing - #825

Merged
robarnold merged 16 commits into
mainfrom
robarnold/codex-fix-devcontainer-base-ci
Aug 4, 2026
Merged

fix(devcontainer-base): validate base updates before publishing#825
robarnold merged 16 commits into
mainfrom
robarnold/codex-fix-devcontainer-base-ci

Conversation

@robarnold

@robarnold robarnold commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Summary

  • revert the devcontainer base from Fedora 45 to Fedora 44
  • make every regular CI job build and load the checked-out devcontainer base locally
  • preserve the existing build, check-format, lint, and kustomization-test check contexts
  • preserve devcontainer registry cache export through Docker's containerd image store
  • validate base-image updates through the full downstream pipeline before publishing
  • publish the exact tested candidate as latest only after validation succeeds
  • keep pull-request builds non-publishing and serialize base-image builds for the same ref

Root cause

The Fedora 45 update moved the devcontainer to Python 3.15. aiohttp did not provide a CPython 3.15 wheel, so the devcontainer build fell back to a source build and failed because no compiler was installed.

The base-image workflow previously published latest as soon as the base itself built. Its pull-request run did not exercise the downstream devcontainer, so the incompatible image reached latest despite green PR checks.

CI behavior

Regular CI assigns the checked-out base a commit-specific tag, builds and loads it on each isolated runner, and passes that tag into the devcontainer build. The required checks therefore validate repository source rather than depending on the already-published latest.

The regular build job enables Docker's containerd image store. That keeps the default daemon-backed builder able to consume the locally loaded candidate while also exporting the shared devcontainer registry cache.

The base-image workflow separately builds one candidate, runs check-format, lint, and the complete kustomization test suite against it, then retags and pushes that exact local image on main and scheduled runs. Pull-request runs never publish.

Validation

  • formatted .github/workflows/checks.yml with the repository's Prettier version
  • parsed the workflow with yq
  • passed git diff --check
  • verified all four job definitions build a local base and still invoke devcontainers/ci
  • verified the regular build log contains both the commit-specific DEVCONTAINER_BASE_TAG and --cache-to ghcr.io/marinatedconcrete/config-devcontainer
  • all nine regular checks passed: build, formatting, lint, and all six kustomization tests
  • candidate-image validation passed the full downstream pipeline; the PR publish step was skipped

Keep the protected branch's existing check contexts running against the known-good Fedora 44 digest. The base-image workflow overrides that default with its explicit candidate image and validates the full pipeline before publishing.
Keep base-image validation focused on the candidate built by the workflow. Fedora registry digests are not a durable fallback because their referenced layers are not retained.
Build and load the checked-out base image on each isolated runner so regular CI never depends on the already-published latest tag. Keep the existing required check contexts while removing the bootstrap failure.
@robarnold
robarnold requested a review from sdwilsh August 3, 2026 04:42
@robarnold
robarnold marked this pull request as ready for review August 3, 2026 04:42
Comment thread .github/workflows/checks.yml
Comment thread .github/workflows/checks.yml
Enable Docker's containerd image store so the daemon-backed builder can consume the locally loaded base and continue exporting the shared devcontainer cache.
@robarnold
robarnold merged commit 5eb834f into main Aug 4, 2026
12 checks passed
@robarnold
robarnold deleted the robarnold/codex-fix-devcontainer-base-ci branch August 4, 2026 03:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants