Skip to content

BUILD/CI: Add ZE null-driver build and smoke check - #11569

Open
yafshar wants to merge 5 commits into
openucx:masterfrom
intel-staging:topic/ze-null-driver-ci
Open

BUILD/CI: Add ZE null-driver build and smoke check#11569
yafshar wants to merge 5 commits into
openucx:masterfrom
intel-staging:topic/ze-null-driver-ci

Conversation

@yafshar

@yafshar yafshar commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

What?

Add a no-GPU CI lane that builds the UCX Level Zero (ZE) backend and performs a runtime smoke check without requiring Intel GPU hardware.

  • Introduce a new ze PR build mode in buildlib/tools/builds.sh
    • Configure with --with-ze
    • Verify HAVE_ZE
    • Build and install UCX
    • Assert that the ZE memory domain (ze_cpy) and transport (ze_copy) enumerate via ucx_info -d
  • Runtime hardware detection follows the same availability pattern as build_cuda and build_rocm
    • If an Intel GPU is present: run real hardware smoke tests
    • Otherwise: fall back to the Level Zero null driver (synthetic device)
    • Optional require_real_ze disables fallback for validation on GPU nodes
  • Add a new builder image buildlib/dockers/ze-builder.Dockerfile
    • Based on intel/oneapi-toolkit:2026.0.0-devel-ubuntu24.04
    • Uses the shipped ZE loader and headers
    • Builds the in-tree Level Zero null driver from a tag matching the loader version (v1.27.0)
    • Installs only the versioned libze_null.so.*
  • Add a new ubuntu2404_ze container resource and x86_64 CI matrix lane
  • Update az_init_modules to return early when Environment Modules are not present

Why?

UCX includes a ZE backend (src/uct/ze) but currently lacks CI coverage, allowing regressions to land undetected.

This change:

  • Adds a --with-ze CI build
  • Provides a basic runtime validation path
  • Enables testing without requiring physical hardware

Real Intel GPU nodes are not available in upstream CI, so the lane uses the
Level Zero loader's null driver (ZE_ENABLE_NULL_DRIVER=1) to provide a
synthetic device for a hardware-independent smoke path.

The null driver is not shipped by the Level Zero release/runtime packages, so
the builder image builds it from Level Zero source at the tag matching the
loader version. Setting ZE_ENABLE_NULL_DRIVER=1 only enables the loader path;
without libze_null.so.* installed, the loader finds no null-driver library and
reports zero devices — which is why the image installs the built driver.

How?

  • The null driver so name is versioned (libze_null.so.<loader_version>)
  • The Dockerfile detects the loader version and builds the matching Level Zero tag
  • Only libze_null.so.* is installed, leaving the system loader intact

build_ze() runtime behavior:

  • Runs ucx_info -d without null driver variables
  • If ze_copy enumerates: use real hardware
  • Otherwise: fall back to the null driver

This ensures:

  • Consistent behavior across environments
  • Automatic use of hardware when available
  • Fully hardware-independent CI

Verification

Validated on a no-GPU x86_64 host:

  • Image build detects loader 1.27.0 and installs libze_null.so.1.27.0
  • zello_world runs successfully under the null driver
  • UCX builds with --with-ze and reports HAVE_ZE=1
  • Real-device probe returns zero devices and triggers fallback
  • ucx_info -d enumerates ze_cpy and ze_copy

CI Image

The builder image is published and referenced directly in main.yml:

  • intel/ucx-ze-builder:oneapi-2026.0.0-l0-1.27.0

The ubuntu2404_ze lane pulls this image directly. A companion source image,
intel/ucx-ze-builder-sources, provides the corresponding source for the open
source components distributed in the builder image.

@yafshar
yafshar force-pushed the topic/ze-null-driver-ci branch from c0c7164 to 12b3804 Compare June 24, 2026 16:47
Add a "ze" PR build mode that configures UCX with --with-ze, verifies
HAVE_ZE is detected, and checks that the ZE memory domain (ze_cpy) and
transport (ze_copy) are enumerated via ucx_info.

Run this build in a new ubuntu2404_ze container derived from the Intel
oneAPI toolkit image. The container provides the Level Zero loader and
headers, and includes the in-tree Level Zero null driver. When no real
Intel GPU is present, the null driver exposes a synthetic device,
allowing ZE build and basic runtime coverage on GPU-less CI workers.

Provide an opt-in require_real_ze path that disables the null fallback
and fails unless a real GPU is detected, for validation on Intel GPU
nodes.

Update az_init_modules to return early when the Environment Modules
system is unavailable, since the public Intel container does not include
it.

Signed-off-by: Yaser Afshar <yaser.afshar@intel.com>
@yafshar
yafshar force-pushed the topic/ze-null-driver-ci branch from a895e8a to 78e5c65 Compare July 29, 2026 16:15
@yafshar
yafshar marked this pull request as ready for review July 29, 2026 16:16
@yafshar

yafshar commented Jul 29, 2026

Copy link
Copy Markdown
Contributor Author

@yosefe @brminich @tvegas1, this PR adds the ZE CI coverage requested in #11218: a ze build mode in builds.sh that configures UCX with --with-ze, verifies HAVE_ZE, and checks that ze_cpy and ze_copy are enumerated by ucx_info.

Regarding the mock-library question, the Level Zero loader's null driver exposes a synthetic device, allowing the lane to run on GPU-less CI workers. The null driver is not included in the Level Zero release packages, so buildlib/dockers/ze-builder.Dockerfile builds it from the source tag matching the loader. The resulting image is published as intel/ucx-ze-builder on Docker Hub and used directly by the ubuntu2404_ze lane.

The build mode can also use a real Intel GPU when the container is given access to one; otherwise it falls back to the null driver. Ready for review.

Comment thread buildlib/pr/main.yml Outdated
Pulling the ZE builder from Docker Hub is subject to anonymous pull
rate limits, which intermittently fail CI. Point the ubuntu2404_ze
container at the internal mirror, matching every other container in
the pipeline, and update the Dockerfile publish note accordingly.

Co-authored-by: Alexey Rivkin <arivkin@nvidia.com>
@yafshar
yafshar requested a review from Alexey-Rivkin August 8, 2026 15:15
@yafshar

yafshar commented Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review @Alexey-Rivkin. CI is fully green (162/162, including the
new ubuntu2404_ze lane) and there are no outstanding comments.

@dpressle when you get a chance, could you take a look? This is CI/build-only
(buildlib/), no changes under src/. Happy to rebase or update if that helps review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants