Skip to content

[TEST] ROCm mlir_aie wheel on Ryzen AI self-hosted runners#3323

Draft
jgmelber wants to merge 6 commits into
mainfrom
test-rocm-wheel-ryzenai
Draft

[TEST] ROCm mlir_aie wheel on Ryzen AI self-hosted runners#3323
jgmelber wants to merge 6 commits into
mainfrom
test-rocm-wheel-ryzenai

Conversation

@jgmelber

@jgmelber jgmelber commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds .github/workflows/testRocmWheelOnRyzenAI.yml to validate the ROCm-built mlir_aie wheel against real Ryzen AI NPU hardware before promoting it into the production mlir-distro pipeline.

  • Build: builds mlir-aie against the current rocm-mlir-distro release wheels (same commit that already passed the (now-closed) [TEST] Add ROCm migration test workflow #3314 migration test), on ubuntu-22.04.
  • Test: pip-installs the resulting wheel exactly as a real IRON user would (not a from-source build), then runs the programming_examples/ Makefile-driven examples on self-hosted NPU runners (aie2-4col, aie2p-8col), via a new utils/run_programming_examples_from_wheel.py that replays each example's *.lit file REQUIRES/RUN metadata directly.

Scope note: test/ and programming_guide/ are intentionally not covered here — their lit suites need a CMake-configured lit.site.cfg.py (confirmed: find_package(AIE CONFIG) standalone-configure fails today because the wheel's AIEConfig.cmake doesn't export imported executable targets like aie-translate that add_lit_testsuite's DEPENDS expects — fixing that is its own, separate CMake-export task). buildAndTestRyzenAI.yml already covers those two suites via a from-source build.

This PR should run and pass before merging the companion "bump MLIR distro to ROCm/llvm-project" PR, per the plan to validate on CI hardware first.

Test plan

  • Manually dispatch this workflow and confirm build_wheel succeeds
  • Confirm test_examples_on_hardware runs on both aie2-4col and aie2p-8col and the programming_examples summary shows passes with no unexpected failures

🤖 Generated with Claude Code

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a dedicated GitHub Actions workflow to build the ROCm-based mlir_aie wheel and validate it by running programming_examples/ directly on self-hosted Ryzen AI NPU runners, using a new helper script that replays *.lit metadata without requiring a CMake/lit build tree.

Changes:

  • Introduces .github/workflows/testRocmWheelOnRyzenAI.yml to build the wheel against rocm-mlir-distro, install it in a fresh venv, and run programming_examples/ on aie2-4col and aie2p-8col hardware runners.
  • Adds utils/run_programming_examples_from_wheel.py to parse REQUIRES/RUN (and XFAIL) from makefile-oriented *.lit files and execute them via run_on_npu.py.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
utils/run_programming_examples_from_wheel.py Implements lit-file parsing and Makefile example execution against a pip-installed wheel on hardware runners.
.github/workflows/testRocmWheelOnRyzenAI.yml New CI workflow to build the ROCm wheel and validate it by running programming examples on Ryzen AI NPUs.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread utils/run_programming_examples_from_wheel.py Outdated
Comment thread utils/run_programming_examples_from_wheel.py Outdated
@jgmelber jgmelber marked this pull request as draft July 13, 2026 15:24
@jgmelber jgmelber changed the title Test ROCm mlir_aie wheel on Ryzen AI self-hosted runners [TEST] ROCm mlir_aie wheel on Ryzen AI self-hosted runners Jul 13, 2026
@jgmelber

Copy link
Copy Markdown
Collaborator Author

Root cause of the `Run programming_examples` failures on both `aie2-4col` and `aie2p-8col`

Both hardware legs fail identically (0/42 and 0/46 passed), overwhelmingly (35/42, 39/46) with:

```
RuntimeError: [aiecc] Compilation failed with exit code 1:
Error: Command failed with exit code -2
Error message: Segmentation fault (core dumped)
Error generating PDI
```

This is a real, hardware-independent packaging bug in this wheel's `bootgen` binary — not a flake or missing-hardware artifact. Diagnosis:

  • Comparing this wheel's `bootgen` against the current officially-released `mlir_aie` wheel's `bootgen`:
    • Official: `ELF ... pie executable` (`ET_DYN`) — runs fine (`bootgen -help` prints the normal banner).
    • This wheel: `ELF ... EXEC (Executable file)` (non-PIE) — segfaults immediately, even under `LD_TRACE_LOADED_OBJECTS=1` (what `ldd` uses internally), i.e. the dynamic loader crashes before `bootgen`'s own code ever runs.
  • `strace` shows the actual mechanism: `SIGSEGV` with `si_code=SEGV_ACCERR` (access-permission violation on a mapped page, not a wild pointer) at address `0x4020c8`.
  • `readelf -l` shows that address falls inside the `DYNAMIC` program header segment, which is mapped `RW` only (no execute permission). Something is jumping there to execute code — the signature of a corrupted GOT/PLT entry or bad relocation.
  • `bootgen` has no dependency on MLIR/LLVM at all (it only links `OpenSSL::SSL` + its own `bootgen-lib`), so the PIE→non-PIE flip has to come from a global CMake/compiler flag that changed when configuring against the ROCm-built MLIR package's exported `LLVMConfig.cmake`/`HandleLLVMOptions.cmake` (both execute once at the top of `cmake configure` and apply to every target in the same build, `bootgen` included) — not anything ROCm-specific in `bootgen`'s own build rules.
  • `auditwheel repair`'s `patchelf`-based RPATH rewrite (needed to point at vendored/renamed OpenSSL libs) appears to mishandle this non-standard `EXEC`-with-dynamic-libs layout, corrupting the binary during that patch — a known-ish `patchelf` weak spot with non-PIE executables, just newly triggered here.

This blocks both this PR and #3326 until fixed. Proposed fix (as a follow-up PR): force `POSITION_INDEPENDENT_CODE ON` on the `bootgen`/`bootgen-lib` CMake targets directly, so they no longer depend on whatever global flag differs in this build — sidesteps the root cause rather than trying to fix `patchelf`/`auditwheel` itself.

jgmelber and others added 2 commits July 13, 2026 17:58
Builds mlir-aie against the current rocm-mlir-distro release, pip-installs
the resulting wheel like a real IRON user would, and runs the
programming_examples/ Makefile-driven examples on self-hosted NPU runners
(aie2-4col / aie2p-8col). This is meant to gate promoting the ROCm-built
MLIR into the production mlir-distro pipeline (see the companion
rocm-llvm-distro-migration PR, which should wait on this one).

test/ and programming_guide/ are out of scope: their lit suites need a
CMake-configured lit.site.cfg.py, which a bare pip install can't produce.
utils/run_programming_examples_from_wheel.py replays each
programming_examples/**/*.lit file's REQUIRES/RUN metadata directly
(matching lit's %S / %run_on_npuN% substitutions) instead of going through
a CMake/lit harness.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Expand %t/%T (lit's per-test scratch path substitutions) the same way
  lit does: %T is the per-test temp directory, %t is a unique path stem
  within it. Without this, lit files like
  programming_examples/ml/mobilenet/run_strix_makefile.lit (which uses
  %t.work) would pass the literal string through instead of a real path.
- Fix inverted XFAIL handling: a command that succeeds under XFAIL is an
  XPASS (lit treats this as a regression signal -- the XFAIL marker is
  now stale), not a normal "expected failure" outcome. Previously this
  script silently folded XPASS into the xfailed bucket, hiding the
  signal. Now tracked separately and treated as a failure for exit-code
  purposes.
- Apply black formatting flagged by CI.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@jgmelber jgmelber force-pushed the test-rocm-wheel-ryzenai branch from 604dfc3 to f092565 Compare July 13, 2026 23:58
jgmelber and others added 4 commits July 13, 2026 20:00
The bootgen PIE fix (#3333) is confirmed working: the shipped bootgen
is now a true PIE binary and no longer segfaults during PDI generation.
That unmasked a second, unrelated failure: most programming_examples/
CMakeLists.txt files require CMake >= 3.30 for their host-code build,
but the self-hosted Ryzen AI runners only have system CMake 3.28.3,
so `Run programming_examples against the wheel` fails uniformly with
"CMake 3.30 or higher is required" once PDI generation stops being the
first thing to fail.

Install cmake via pip into the test venv instead of relying on the
runner's system package -- same version pin already used in
python/requirements_dev.txt for the from-source build path. Venv
activation puts wheel-test-venv/bin ahead of the system PATH, so this
shadows the outdated system cmake without needing to touch anything on
the runners themselves.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…r cmake/ninja

The previous commit added an ad hoc, unhashed `pip install cmake` that
only covered cmake, not ninja (also needed to compile programming_examples/
host code) and didn't match how this exact same problem is already solved
elsewhere in the repo. buildRyzenWheels.yml's programming_examples smoke
test job installs the runtime env via env_install.sh, then layers on
`pip install --require-hashes -r python/requirements_dev.lock` for the
build tooling — mirror that exactly instead of inventing a new mechanism.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Every hardware-touching example failed on both self-hosted runners with
mmap(...) failed (err=-11): Resource temporarily unavailable, from the very
first XRT device open. buildAndTestRyzenAI.yml and buildRyzenWheels.yml both
work around the runners' default RLIMIT_MEMLOCK with `prlimit -l unlimited`
before touching the NPU; this new workflow was missing that step.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…red optional dep

Three examples still failed after the RLIMIT_MEMLOCK fix, none of them NPU
issues:
- mobilenet/run_strix_makefile.lit imports torch via mb_utils.py but its
  REQUIRES doesn't declare torch, so the opencv/torch REQUIRES-based skip
  never triggered.
- tiling_exploration/introduction needs jupyter+nbconvert (notebook tutorial)
  and per_tile/tile_group pipe through FileCheck -- both are dev-toolchain
  binaries a bare `pip install mlir_aie` user never gets, and neither has a
  REQUIRES tag to key off of.

Rather than special-casing each path, recognize the failure signature
(ModuleNotFoundError / "command not found") generically and report it as
skipped instead of failed, the same bucket the existing REQUIRES-based
opencv/torch/chess checks use.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants