[TEST] ROCm mlir_aie wheel on Ryzen AI self-hosted runners#3323
[TEST] ROCm mlir_aie wheel on Ryzen AI self-hosted runners#3323jgmelber wants to merge 6 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a dedicated GitHub Actions workflow to build the ROCm-based mlir_aie wheel and validate it by running programming_examples/ directly on self-hosted Ryzen AI NPU runners, using a new helper script that replays *.lit metadata without requiring a CMake/lit build tree.
Changes:
- Introduces
.github/workflows/testRocmWheelOnRyzenAI.ymlto build the wheel againstrocm-mlir-distro, install it in a fresh venv, and runprogramming_examples/onaie2-4colandaie2p-8colhardware runners. - Adds
utils/run_programming_examples_from_wheel.pyto parseREQUIRES/RUN(andXFAIL) from makefile-oriented*.litfiles and execute them viarun_on_npu.py.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| utils/run_programming_examples_from_wheel.py | Implements lit-file parsing and Makefile example execution against a pip-installed wheel on hardware runners. |
| .github/workflows/testRocmWheelOnRyzenAI.yml | New CI workflow to build the ROCm wheel and validate it by running programming examples on Ryzen AI NPUs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Root cause of the `Run programming_examples` failures on both `aie2-4col` and `aie2p-8col`Both hardware legs fail identically (0/42 and 0/46 passed), overwhelmingly (35/42, 39/46) with: ``` This is a real, hardware-independent packaging bug in this wheel's `bootgen` binary — not a flake or missing-hardware artifact. Diagnosis:
This blocks both this PR and #3326 until fixed. Proposed fix (as a follow-up PR): force `POSITION_INDEPENDENT_CODE ON` on the `bootgen`/`bootgen-lib` CMake targets directly, so they no longer depend on whatever global flag differs in this build — sidesteps the root cause rather than trying to fix `patchelf`/`auditwheel` itself. |
c9add9a to
604dfc3
Compare
Builds mlir-aie against the current rocm-mlir-distro release, pip-installs the resulting wheel like a real IRON user would, and runs the programming_examples/ Makefile-driven examples on self-hosted NPU runners (aie2-4col / aie2p-8col). This is meant to gate promoting the ROCm-built MLIR into the production mlir-distro pipeline (see the companion rocm-llvm-distro-migration PR, which should wait on this one). test/ and programming_guide/ are out of scope: their lit suites need a CMake-configured lit.site.cfg.py, which a bare pip install can't produce. utils/run_programming_examples_from_wheel.py replays each programming_examples/**/*.lit file's REQUIRES/RUN metadata directly (matching lit's %S / %run_on_npuN% substitutions) instead of going through a CMake/lit harness. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Expand %t/%T (lit's per-test scratch path substitutions) the same way lit does: %T is the per-test temp directory, %t is a unique path stem within it. Without this, lit files like programming_examples/ml/mobilenet/run_strix_makefile.lit (which uses %t.work) would pass the literal string through instead of a real path. - Fix inverted XFAIL handling: a command that succeeds under XFAIL is an XPASS (lit treats this as a regression signal -- the XFAIL marker is now stale), not a normal "expected failure" outcome. Previously this script silently folded XPASS into the xfailed bucket, hiding the signal. Now tracked separately and treated as a failure for exit-code purposes. - Apply black formatting flagged by CI. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
604dfc3 to
f092565
Compare
The bootgen PIE fix (#3333) is confirmed working: the shipped bootgen is now a true PIE binary and no longer segfaults during PDI generation. That unmasked a second, unrelated failure: most programming_examples/ CMakeLists.txt files require CMake >= 3.30 for their host-code build, but the self-hosted Ryzen AI runners only have system CMake 3.28.3, so `Run programming_examples against the wheel` fails uniformly with "CMake 3.30 or higher is required" once PDI generation stops being the first thing to fail. Install cmake via pip into the test venv instead of relying on the runner's system package -- same version pin already used in python/requirements_dev.txt for the from-source build path. Venv activation puts wheel-test-venv/bin ahead of the system PATH, so this shadows the outdated system cmake without needing to touch anything on the runners themselves. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…r cmake/ninja The previous commit added an ad hoc, unhashed `pip install cmake` that only covered cmake, not ninja (also needed to compile programming_examples/ host code) and didn't match how this exact same problem is already solved elsewhere in the repo. buildRyzenWheels.yml's programming_examples smoke test job installs the runtime env via env_install.sh, then layers on `pip install --require-hashes -r python/requirements_dev.lock` for the build tooling — mirror that exactly instead of inventing a new mechanism. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Every hardware-touching example failed on both self-hosted runners with mmap(...) failed (err=-11): Resource temporarily unavailable, from the very first XRT device open. buildAndTestRyzenAI.yml and buildRyzenWheels.yml both work around the runners' default RLIMIT_MEMLOCK with `prlimit -l unlimited` before touching the NPU; this new workflow was missing that step. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…red optional dep Three examples still failed after the RLIMIT_MEMLOCK fix, none of them NPU issues: - mobilenet/run_strix_makefile.lit imports torch via mb_utils.py but its REQUIRES doesn't declare torch, so the opencv/torch REQUIRES-based skip never triggered. - tiling_exploration/introduction needs jupyter+nbconvert (notebook tutorial) and per_tile/tile_group pipe through FileCheck -- both are dev-toolchain binaries a bare `pip install mlir_aie` user never gets, and neither has a REQUIRES tag to key off of. Rather than special-casing each path, recognize the failure signature (ModuleNotFoundError / "command not found") generically and report it as skipped instead of failed, the same bucket the existing REQUIRES-based opencv/torch/chess checks use. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Summary
Adds
.github/workflows/testRocmWheelOnRyzenAI.ymlto validate the ROCm-builtmlir_aiewheel against real Ryzen AI NPU hardware before promoting it into the productionmlir-distropipeline.rocm-mlir-distrorelease wheels (same commit that already passed the (now-closed) [TEST] Add ROCm migration test workflow #3314 migration test), onubuntu-22.04.programming_examples/Makefile-driven examples on self-hosted NPU runners (aie2-4col,aie2p-8col), via a newutils/run_programming_examples_from_wheel.pythat replays each example's*.litfileREQUIRES/RUNmetadata directly.Scope note:
test/andprogramming_guide/are intentionally not covered here — their lit suites need a CMake-configuredlit.site.cfg.py(confirmed:find_package(AIE CONFIG)standalone-configure fails today because the wheel'sAIEConfig.cmakedoesn't export imported executable targets likeaie-translatethatadd_lit_testsuite'sDEPENDSexpects — fixing that is its own, separate CMake-export task).buildAndTestRyzenAI.ymlalready covers those two suites via a from-source build.This PR should run and pass before merging the companion "bump MLIR distro to ROCm/llvm-project" PR, per the plan to validate on CI hardware first.
Test plan
build_wheelsucceedstest_examples_on_hardwareruns on bothaie2-4colandaie2p-8coland theprogramming_examplessummary shows passes with no unexpected failures🤖 Generated with Claude Code