Fix unified pip devcontainer UCX base tags - #766
Merged
msarahan merged 4 commits intoSep 9, 2026
Conversation
msarahan
requested review from
KyleFromNVIDIA
and removed request for
a team
September 9, 2026 00:00
msarahan
added
non-breaking
msarahan
force-pushed
the
codex/fix-unified-ucx-1.21
branch
from
September 9, 2026 00:38
9c8c031 to
2286e75
Compare
The PyPI OpenMPI wheel bundles UCX 1.20 libraries. Unified pip containers now use system UCX 1.21, so installing that wheel creates a mixed UCX runtime and leaves libucp with unresolved 1.21 symbols during C++ links.\n\nExclude and uninstall the wheel whenever system UCX is preferred. The base image already supplies OpenMPI linked against that same system UCX, preserving one internally consistent MPI/UCX stack even for reused virtual environments.
Contributor
Author
|
Posted by Codex (gpt-5.6-sol) on Michael Sarahan's behalf; treat this message as LLM-generated. /ok to test 5b72ce4 |
The CUDA feature exports an incomplete profiler-only system toolkit for conda devcontainers. Conda's tileiras then honors that CUDA_HOME instead of using the matching compiler libraries installed in the environment, causing every TileIR version probe and cuVS kernel export to fail.\n\nAllow the CUDA profile to accept an explicit package-managed toolkit root, and select the rapids conda environment only for the CUDA 13.3 conda configuration that installs tileiras. Other CUDA and package-manager combinations retain the system default.
Contributor
Author
|
Posted by Codex (gpt-5.6-sol) on Michael Sarahan's behalf; treat this message as LLM-generated. /ok to test 93801e3 |
Contributor
Author
|
Posted by Codex (gpt-5.6-sol) on Michael Sarahan's behalf; treat this message as LLM-generated. /ok to test |
bdice
reviewed
Sep 9, 2026
bdice
left a comment
bdice
left a comment
Contributor
There was a problem hiding this comment.
The ucx 1.21 changes are correct, and are also included in #767. I approved that PR.
I am not super confident about the CUDA_HOME environment changes here. For awareness, @jameslamb is looking at some cuda-tile modifications in rapidsai/build-planning#324.
Contributor
|
/ok to test 93801e3 |
Contributor
|
@msarahan The AI disclaimer preface was preventing |
cuTile dependencies should be declared by the project that needs them, while tileiras comes from the system CUDA Toolkit selected for that build. Redirecting CUDA_HOME for the entire conda devcontainer hides that dependency boundary and changes unrelated CUDA consumers, so restore the feature's normal system-toolkit environment.
Member
^ also notice your AI-generated description linked to #324 (random closed PR from 2 years ago) when I'm pretty sure the intention was rapidsai/build-planning#324. |
Contributor
Author
|
@jameslamb you're absolutely right. That's kind of a language ambiguity that github turns into a misleading link. It is linguistically correct, but not technically. Thanks for pointing it out. |
bdice
approved these changes
Sep 9, 2026
Contributor
Author
|
/merge |
trxcllnt
reviewed
Sep 9, 2026
trxcllnt
left a comment
trxcllnt
left a comment
Collaborator
There was a problem hiding this comment.
Need to update the feature version in features/src/rapids-build-utils/devcontainer-feature.json, otherwise the new changes won't be published to ghcr.io
Contributor
Author
|
Followed up with #769 |
jrhemstad
pushed a commit
to jrhemstad/devcontainers
that referenced
this pull request
Sep 10, 2026
rapidsai#766 did not include the necessary bump
msarahan
added a commit
that referenced
this pull request
Sep 10, 2026
**Posted by Codex (GPT-5) on Michael Sarahan's behalf. Treat this message as LLM-generated.** ## Why The 26.12 version update triggered the release workflow before `v26.12.00a` existed. Image producers derive their namespace from the most recent reachable tag, so they published successful `26.10-*` images while the newly updated unified devcontainers requested `26.12-*` images. The alpha tag appeared nine minutes after the workflow began. A later retry of failed jobs could not recover because GitHub did not rerun the already-successful producer jobs. This creates an avoidable bootstrap dependency: the version-update workflow needs to publish the new namespace before the tag that currently defines that namespace can be created. The committed `VERSION` file is already updated atomically with the consumer definitions and is therefore the appropriate source of truth for producer tags. The incident also exposed a second class of mismatch: `matrix.yml` can change an image component version without the checked-in unified devcontainers changing their `BASE` references. Without validation, that drift is only discovered after all producer builds complete and consumers attempt to pull an image that will never be published. PR #766 applies the UCX alignment to `release/26.10` for forward-merging. This branch currently contains the equivalent main-branch alignment; that portion will disappear from this PR's diff after the forward merge reaches `main`. ## Changes - derive Linux and Windows producer namespaces from the committed `VERSION` file instead of `git describe` - validate that `VERSION` uses `YY.MM.PP` format - reconstruct all publishable Linux image tags from `matrix.yml`, including OS-qualified tags and OS-free aliases - reject any checked-in RAPIDS unified devcontainer `BASE` that the current matrix cannot produce - run that validation in pre-commit and as a fatal release prerequisite before Linux producers and downstream unified devcontainers run - add ShellCheck coverage for the release-bootstrap scripts touched here ## Why the previous sequence cannot recur After these changes, merging a version update changes both producer and consumer namespaces through the same committed `VERSION` value. The producer no longer depends on whether the alpha tag existed when the workflow checkout occurred. The release also stops before building images if a consumer base does not correspond to the current matrix, so component-version drift is reported directly rather than surfacing as a late missing-manifest error. ## Validation - `pre-commit run --all-files` - zizmor - devcontainer base-image validation - ShellCheck for the release-bootstrap scripts - positive validation against the 26.12 / UCX 1.21 configuration - negative validation with a deliberate UCX 1.19 base, which failed and identified the affected file and unproducible tag - `git diff --check` Incident: https://github.com/rapidsai/devcontainers/actions/runs/34280170418/job/102279620120 --------- Co-authored-by: Paul Taylor <178183+trxcllnt@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Posted by Codex (gpt-5.6-sol) on Michael Sarahan's behalf; treat this message as LLM-generated.
The
release/26.10image matrix builds UCX 1.21 (as of NVIDIA/devcontainers#756), but its checked-in unified pip devcontainers still requested UCX 1.19 base images. Those tags cannot be produced by the current branch matrix, so this PR updates the CUDA 12.9 and 13.3 pip bases to UCX 1.21.The first full matrix exposed a second UCX compatibility problem hidden by the old configuration. The PyPI OpenMPI wheel bundles UCX 1.20 libraries. Installing that wheel while RAPIDS explicitly selects system UCX 1.21 creates a mixed runtime and leaves new
libucpsymbols unresolved during C++ links. When system UCX is preferred, the pip environment now excludes and removes the PyPI OpenMPI wheel, including from reused virtual environments. The base image already supplies OpenMPI built against its system UCX, so MPI and UCX remain one consistent stack.This PR deliberately does not change
CUDA_HOMEto address the separate cuTile failures seen in the first run. RAPIDS build-planning issue rapidsai/build-planning#324 establishes the intended dependency boundary: projects declare thecuda-tilePython build dependency, whiletileirascomes from the system CUDA Toolkit selected for the build. The corresponding work is in rapidsai/cugraph#5646 and NVIDIA/cuvs#2564. Redirecting the entire conda devcontainer to a package-managed toolkit would hide that dependency problem and change the toolkit seen by unrelated CUDA consumers.Related dependency work: