[CI] 26.12 pip devcontainer builds fail to pull base images

[csadorf]

Summary

The 26.12 CUDA 12.9 and CUDA 13.3 pip devcontainer builds failed on both amd64 and arm64. Docker reported that no manifest was available for the referenced rapidsai/devcontainers base images. The corresponding conda devcontainer builds completed successfully.

Failing test/component: Build devcontainers / pip variants

Failure observed in:

• https://github.com/NVIDIA/cuml/actions/runs/34321012514
• https://github.com/NVIDIA/cuml/actions/runs/34321012514/job/102367427349
• https://github.com/NVIDIA/cuml/actions/runs/34321012514/job/102367427361
• https://github.com/NVIDIA/cuml/actions/runs/34321012514/job/102367427430
• https://github.com/NVIDIA/cuml/actions/runs/34321012514/job/102367427464

Environment

• OS: Ubuntu 24.04.4 LTS runner
• Python version: N/A
• CUDA version: 12.9 and 13.3
• GPU: CPU-only devcontainer build
• Dependencies: rapidsai/devcontainers:26.12-cpp-cuda12.9-ucx1.19.0-openmpi5.0.10, rapidsai/devcontainers:26.12-cpp-cuda13.3-ucx1.19.0-openmpi5.0.10

Test Details

• Test file: .devcontainer/cuda12.9-pip/devcontainer.json, .devcontainer/cuda13.3-pip/devcontainer.json

• Test name: Build devcontainers / build (..., pip)

• Error message:

  Error fetching image details: No manifest found for docker.io/rapidsai/devcontainers:26.12-cpp-cuda12.9-ucx1.19.0-openmpi5.0.10.
  Command failed: docker pull rapidsai/devcontainers:26.12-cpp-cuda12.9-ucx1.19.0-openmpi5.0.10
  Failed to build ghcr.io/nvidia/cuml/devcontainer:26.12-cuda12.9-pip
  Process completed with exit code 1.
  

  The CUDA 13.3 jobs report the same messages for rapidsai/devcontainers:26.12-cpp-cuda13.3-ucx1.19.0-openmpi5.0.10 and ghcr.io/nvidia/cuml/devcontainer:26.12-cuda13.3-pip.

Root Cause Analysis

This likely resulted from the 26.12.00 version update on main during the 26.10 burn-down transition. The update caused the pip devcontainer builds to reference 26.12 base-image tags before those tags were available on Docker Hub.

Both exact tags still returned HTTP 404 from the Docker Hub registry when checked after the CI run:

• rapidsai/devcontainers:26.12-cpp-cuda12.9-ucx1.19.0-openmpi5.0.10
• rapidsai/devcontainers:26.12-cpp-cuda13.3-ucx1.19.0-openmpi5.0.10

[csadorf]

This is expected to be resolved by rapidsai/devcontainers#767. rapidsai/devcontainers#766 contains the corresponding change for release/26.10.