BUILD/CI: Add ZE null-driver build and smoke check - #11569
Conversation
c0c7164 to
12b3804
Compare
Add a "ze" PR build mode that configures UCX with --with-ze, verifies HAVE_ZE is detected, and checks that the ZE memory domain (ze_cpy) and transport (ze_copy) are enumerated via ucx_info. Run this build in a new ubuntu2404_ze container derived from the Intel oneAPI toolkit image. The container provides the Level Zero loader and headers, and includes the in-tree Level Zero null driver. When no real Intel GPU is present, the null driver exposes a synthetic device, allowing ZE build and basic runtime coverage on GPU-less CI workers. Provide an opt-in require_real_ze path that disables the null fallback and fails unless a real GPU is detected, for validation on Intel GPU nodes. Update az_init_modules to return early when the Environment Modules system is unavailable, since the public Intel container does not include it. Signed-off-by: Yaser Afshar <yaser.afshar@intel.com>
a895e8a to
78e5c65
Compare
|
@yosefe @brminich @tvegas1, this PR adds the ZE CI coverage requested in #11218: a Regarding the mock-library question, the Level Zero loader's null driver exposes a synthetic device, allowing the lane to run on GPU-less CI workers. The null driver is not included in the Level Zero release packages, so The build mode can also use a real Intel GPU when the container is given access to one; otherwise it falls back to the null driver. Ready for review. |
Pulling the ZE builder from Docker Hub is subject to anonymous pull rate limits, which intermittently fail CI. Point the ubuntu2404_ze container at the internal mirror, matching every other container in the pipeline, and update the Dockerfile publish note accordingly. Co-authored-by: Alexey Rivkin <arivkin@nvidia.com>
|
Thanks for the review @Alexey-Rivkin. CI is fully green (162/162, including the @dpressle when you get a chance, could you take a look? This is CI/build-only |
What?
Add a no-GPU CI lane that builds the UCX Level Zero (ZE) backend and performs a runtime smoke check without requiring Intel GPU hardware.
zePR build mode inbuildlib/tools/builds.sh--with-zeHAVE_ZEze_cpy) and transport (ze_copy) enumerate viaucx_info -dbuild_cudaandbuild_rocmrequire_real_zedisables fallback for validation on GPU nodesbuildlib/dockers/ze-builder.Dockerfileintel/oneapi-toolkit:2026.0.0-devel-ubuntu24.04v1.27.0)libze_null.so.*ubuntu2404_zecontainer resource and x86_64 CI matrix laneaz_init_modulesto return early when Environment Modules are not presentWhy?
UCX includes a ZE backend (
src/uct/ze) but currently lacks CI coverage, allowing regressions to land undetected.This change:
--with-zeCI buildReal Intel GPU nodes are not available in upstream CI, so the lane uses the
Level Zero loader's null driver (
ZE_ENABLE_NULL_DRIVER=1) to provide asynthetic device for a hardware-independent smoke path.
The null driver is not shipped by the Level Zero release/runtime packages, so
the builder image builds it from Level Zero source at the tag matching the
loader version. Setting
ZE_ENABLE_NULL_DRIVER=1only enables the loader path;without
libze_null.so.*installed, the loader finds no null-driver library andreports zero devices — which is why the image installs the built driver.
How?
libze_null.so.<loader_version>)libze_null.so.*is installed, leaving the system loader intactbuild_ze()runtime behavior:ucx_info -dwithout null driver variablesze_copyenumerates: use real hardwareThis ensures:
Verification
Validated on a no-GPU x86_64 host:
1.27.0and installslibze_null.so.1.27.0zello_worldruns successfully under the null driver--with-zeand reportsHAVE_ZE=1ucx_info -denumeratesze_cpyandze_copyCI Image
The builder image is published and referenced directly in
main.yml:intel/ucx-ze-builder:oneapi-2026.0.0-l0-1.27.0The
ubuntu2404_zelane pulls this image directly. A companion source image,intel/ucx-ze-builder-sources, provides the corresponding source for the opensource components distributed in the builder image.