Skip to content

Prevent release image bootstrap mismatches - #767

Merged
msarahan merged 3 commits into
rapidsai:release/26.10from
msarahan:codex/version-bootstrap-preflight
Sep 10, 2026
Merged

Prevent release image bootstrap mismatches#767
msarahan merged 3 commits into
rapidsai:release/26.10from
msarahan:codex/version-bootstrap-preflight

Conversation

@msarahan

@msarahan msarahan commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Posted by Codex (GPT-5) on Michael Sarahan's behalf. Treat this message as LLM-generated.

Why

The 26.12 version update triggered the release workflow before v26.12.00a existed. Image producers derive their namespace from the most recent reachable tag, so they published successful 26.10-* images while the newly updated unified devcontainers requested 26.12-* images. The alpha tag appeared nine minutes after the workflow began. A later retry of failed jobs could not recover because GitHub did not rerun the already-successful producer jobs.

This creates an avoidable bootstrap dependency: the version-update workflow needs to publish the new namespace before the tag that currently defines that namespace can be created. The committed VERSION file is already updated atomically with the consumer definitions and is therefore the appropriate source of truth for producer tags.

The incident also exposed a second class of mismatch: matrix.yml can change an image component version without the checked-in unified devcontainers changing their BASE references. Without validation, that drift is only discovered after all producer builds complete and consumers attempt to pull an image that will never be published.

PR #766 applies the UCX alignment to release/26.10 for forward-merging. This branch currently contains the equivalent main-branch alignment; that portion will disappear from this PR's diff after the forward merge reaches main.

Changes

  • derive Linux and Windows producer namespaces from the committed VERSION file instead of git describe
  • validate that VERSION uses YY.MM.PP format
  • reconstruct all publishable Linux image tags from matrix.yml, including OS-qualified tags and OS-free aliases
  • reject any checked-in RAPIDS unified devcontainer BASE that the current matrix cannot produce
  • run that validation in pre-commit and as a fatal release prerequisite before Linux producers and downstream unified devcontainers run
  • add ShellCheck coverage for the release-bootstrap scripts touched here

Why the previous sequence cannot recur

After these changes, merging a version update changes both producer and consumer namespaces through the same committed VERSION value. The producer no longer depends on whether the alpha tag existed when the workflow checkout occurred. The release also stops before building images if a consumer base does not correspond to the current matrix, so component-version drift is reported directly rather than surfacing as a late missing-manifest error.

Validation

  • pre-commit run --all-files
    • zizmor
    • devcontainer base-image validation
    • ShellCheck for the release-bootstrap scripts
  • positive validation against the 26.12 / UCX 1.21 configuration
  • negative validation with a deliberate UCX 1.19 base, which failed and identified the affected file and unproducible tag
  • git diff --check

Incident: https://github.com/rapidsai/devcontainers/actions/runs/34280170418/job/102279620120

@msarahan
msarahan requested a review from a team as a code owner September 9, 2026 00:06
@msarahan
msarahan requested review from bdice and removed request for a team September 9, 2026 00:06
@msarahan msarahan added non-breaking Introduces a non-breaking change improvement Improves an existing functionality labels Sep 9, 2026

@bdice bdice left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with all of these changes.

@msarahan
msarahan changed the base branch from main to release/26.10 September 9, 2026 16:25
@msarahan
msarahan force-pushed the codex/version-bootstrap-preflight branch from 51a73f6 to d8fc989 Compare September 9, 2026 16:29
@msarahan

Copy link
Copy Markdown
Contributor Author

/merge

@msarahan
msarahan merged commit 30857e4 into rapidsai:release/26.10 Sep 10, 2026
397 of 407 checks passed
@msarahan
msarahan deleted the codex/version-bootstrap-preflight branch September 10, 2026 17:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

improvement Improves an existing functionality non-breaking Introduces a non-breaking change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants