Peano lit enablement No1

[thomthehound]

This updates the lit configuration files (and affected test files) so that AIE/NPU tests can run with the open-source Peano backend. These changes are compatible with (and were tested on) Windows, but are not exclusive to it. Running the tests with the Peano backend should now be possible on Linux as well.

Highlights:

• Detect Peano early enough for Ryzen AI/XRT feature gating, so NPU tests are not incorrectly disabled just because Vitis/AIETOOLS components are absent.
• Gate npu-xrt tests on lit features (peano, chess, aietools_aie2, aietools_aie2p) instead of raw Vitis state.
• Add %backend_flags substitution that keeps the existing Chess path when Chess is available, but automatically expands to --no-xchesscc --no-xbridge when only Peano is present.
• Add cross-platform host compiler/link substitutions, replacing Linux-only clang ... -lrt -lstdc++ command lines with %host_clang and %host_link_flags.
• Harden Windows execution by avoiding POSIX-isms such as |&, stdin echo pipelines, Makefile-based tests, and extensionless executables.
• Fix CMake configuration needed by the lit flow, including defaulting Chess tests only if/after Vitis detection and correcting the LibXAIE_${AIE_RUNTIME_TEST_TARGET}_DIR variable.

This is intended as lit enablement only. Experimentation has demonstrated some separate Python host runtime ownership/lifetime quirks -- at least on Windows -- that still require additional work.

---

Test command conventions

I am recommending several new cross-platform hygiene standards here for AIE/NPU test commands. Ideally, these can be extended to guides, examples, and other parts of the tree later. My intent is two-fold:

• 1. keep tests from depending on PATH, POSIX shell behavior, or Linux-only linker flags and
• 2. avoid the need to push commits to hundreds of individual files every time something like a backend flag changes (for instance, the recent hardening around --no-xbridge as a further example in addition to the present case).

For example, tests should prefer:

• %aiecc %backend_flags ... for compiler-driver invocations
• %host_clang ... %host_link_flags ... for host test binaries
• %PYTHON ... for Python scripts
• explicit .exe names for host executables used by lit (this is safe on Linux and required for Windows)
• lit features such as peano, chess, aietools_aie2, aietools_aie2p, and xrt for gating

And, in future tests, guides, examples, and scripts, etc., it would be best to avoid:

• bare aiecc or clang
• aiecc.py (in most cases)
• hard-coded --no-xchesscc --no-xbridge (%backend_flags supplies this when needed)
• hard-coded -lstdc++ / -lrt (%host_link_flags supplies this)
• |&, stdin echo pipelines, Makefile-only test drivers, and extensionless executable assumptions

The shared substitutions used by the files touched in this PR are defined in lit_config_helpers.py.

---

Reproduction

On Windows, the tests may be exercised locally using the following PowerShell commands. The absolute path of an extracted copy of MLIR must be supplied for $mlir:

# Setup env from fresh clone
python utils/iron_setup.py
python utils/iron_setup.py env --shell pwsh | iex 

# Configure lit tests
$mlir = "F:\dev\mlir"      # Replace with root containing lib\cmake\mlir and lib\cmake\llvm
$xrt = "C:\Xilinx\XRT"     # Adjust if your XRT SDK was extracted elsewhere
$prefix = "$PWD\build_lit\install"

cmake -S . -B build_lit -G Ninja `
  -DCMAKE_BUILD_TYPE=Release `
  -DCMAKE_INSTALL_PREFIX="$prefix" `
  -DMLIR_DIR="$mlir\lib\cmake\mlir" `
  -DLLVM_DIR="$mlir\lib\cmake\llvm" `
  -DPEANO_INSTALL_DIR="$env:PEANO_INSTALL_DIR" `
  -DXRT_ROOT="$xrt" `
  -DCMAKE_PREFIX_PATH="$xrt" `
  -DLLVM_EXTERNAL_LIT=lit `
  -DAIE_COMPILER=NONE `
  -DAIE_LINKER=NONE `
  -DHOST_COMPILER=NONE

# Build needed testing tools
cmake --build .\build_lit

# Run test sweep (parallel)
lit -sv --show-unsupported --time-tests --timeout 600 .\build_lit\test 2>&1 |
  Tee-Object -FilePath .\lit_test_sweep.log

And the equivalent cmd commands from an x64 Native Tools Command Prompt:

python utils\iron_setup.py
python utils\iron_setup.py env --shell cmd > "%TEMP%\iron_env.bat" && call "%TEMP%\iron_env.bat"

set "mlir=F:\dev\mlir"
set "xrt=C:\Xilinx\XRT"
set "prefix=%CD%\build_lit\install"

cmake -S . -B build_lit -G Ninja ^
  -DCMAKE_BUILD_TYPE=Release ^
  -DCMAKE_INSTALL_PREFIX="%prefix%" ^
  -DMLIR_DIR="%mlir%\lib\cmake\mlir" ^
  -DLLVM_DIR="%mlir%\lib\cmake\llvm" ^
  -DPEANO_INSTALL_DIR="%PEANO_INSTALL_DIR%" ^
  -DXRT_ROOT="%xrt%" ^
  -DCMAKE_PREFIX_PATH="%xrt%" ^
  -DLLVM_EXTERNAL_LIT=lit ^
  -DAIE_COMPILER=NONE ^
  -DAIE_LINKER=NONE ^
  -DHOST_COMPILER=NONE

cmake --build .\build_lit

lit -sv --show-unsupported --time-tests --timeout 600 .\build_lit\test > .\lit_test_sweep.log 2>&1
type .\lit_test_sweep.log

cmd does not have an equivalent to tee, so this will appear to hang for several minutes as the log is being written.

(!! IMPORTANT: You must be running Python 3.13 for this to work on Windows, because that is the only version supported by pyxrt.pyd in the XRT SDK !!)

---

Test summary (tail):

********************
Failed Tests (20):
  AIE_TEST :: npu-xrt/add_one_objFifo/run.lit
  AIE_TEST :: npu-xrt/loadpdi/run.lit
  AIE_TEST :: npu-xrt/reconfigure_loadpdi/run_loadpdi.lit
  AIE_TEST :: npu-xrt/reconfigure_loadpdi/run_write32s.lit
  AIE_TEST :: npu-xrt/reconfigure_loadpdi_persistent_memtile/run_loadpdi.lit
  AIE_TEST :: npu-xrt/reconfigure_loadpdi_persistent_memtile/run_write32s.lit
  AIE_TEST :: npu-xrt/scratchpad_regwrite/run.lit
  AIE_TEST :: python-concurrency/jit_parallel_compilation.py
  AIE_TEST :: python/npu-xrt/test_algorithms.py
  AIE_TEST :: python/npu-xrt/test_cached_xrt_runtime.py
  AIE_TEST :: python/npu-xrt/test_cached_xrt_runtime_insts.py
  AIE_TEST :: python/npu-xrt/test_compile_cache_functionality.py
  AIE_TEST :: python/npu-xrt/test_jit_compilation.py
  AIE_TEST :: python/npu-xrt/test_jit_extern_functions.py
  AIE_TEST :: python/npu-xrt/test_jit_extern_functions_inside_jit.py
  AIE_TEST :: python/npu-xrt/test_jit_placed_style.py
  AIE_TEST :: python/npu-xrt/test_jit_trace.py
  AIE_TEST :: python/npu-xrt/test_jit_two_extern_functions.py
  AIE_TEST :: python/npu-xrt/test_jit_utils.py
  AIE_TEST :: python/npu-xrt/test_tensor.py


Testing Time: 205.99s

Total Discovered Tests: 1277
  Unsupported      : 430 (33.67%)
  Passed           : 822 (64.37%)
  Expectedly Failed:   5 (0.39%)
  Failed           :  20 (1.57%)

At this point, the remaining failures may be triaged into the following four categories, all of which are distinct from lit test configuration issues:

1. Full-ELF/XRT coverage

   Six (6) failures are in loadpdi, reconfiguration, and scratchpad-register-write tests that exercise full-ELF attachment through XRT. As tested on Windows, attaching most ELFs to a hw_context works fine, but these particular kinds of full-ELFs do not yet appear to be supported end-to-end, and I suspect that is because they are written in such a way that they touch the firmware. These tests are left ungated for now, but I think it is unlikely they will ever fully work in a signed-driver environment.

2. Python host-runtime ownership/lifetime behavior

   Twelve (12) tests fail due to object lifetime ordering concerning hw_context-attached resources. Specifically, the Windows runtime is sensitive to destroying a hw_context before attached objects, such as BOs, which usually causes a seg-fault. These are correctable through hardening lifetime/ownership in hostruntime.py. I've prototyped that separately and will submit the fixes if/after this PR is merged. Frankly, the destruction order (attachments first, hw_context last) that Windows forces makes more intuitive sense to me, anyway.

3. Context-cache stress behavior

   test_cached_xrt_runtime.py still has one (1) failure involving the creation of an additional hw_context when the configured cache limit is reached. This likely needs separate investigation of the cache eviction semantics.

4. Remaining generated-code/runtime correctness issue

   test_algorithms.py still has one (1) scalar float32 add correctness failure, where-in one 16-element ObjectFifo/data-tile region (corresponding to output entries 96-111) is observed as zeroed. The root cause of this is not yet clear to me, but may involve either Peano-specific code-generation or our own runtime memory handling. In this specific case, the affected ObjectFifo buffer for data-tile 6 is allocated adjacent to the worker stack region, and my testing has indicated that increasing stack_size to 0x0800 (from the default 0x0400) resolves the symptom but not the underlying cause. Altering the instruction to multiply, which uses libcalls, also prevents the corruption. This is consistent with (but potentially not exclusive to) a stack-pressure or spill-related mechanism: Peano’s AIE2P lowering custom-legalizes scalar float32 adds/subs through vector floating-point operations, which use zero-padded ACC2048 registers (and corresponding stack spill/reload machinery). However, other mechanisms cannot yet be ruled out. That is as far as I got on this one. It's a stumper.

6 + 12 + 1 + 1 = 20
All failures are accounted.

[thomthehound]

Converting this to draft. The new scratchpad test introduced a few hours ago is giving me trouble parsing the new custom op.

[thomthehound]

I think I have it nailed down now. The wheel wasn't as fresh as I expected it to be, so I needed to recompile aiecc. Interesting. I thought we distributed new wheels with each merge.

Regardless, looking at the new test gave me another idea to implement while I futzed around. Just a few more hours and I'll have this ready for review.

[thomthehound]

• centralize tool substitutions
• exercise aiecc executable backend where appropriate (instead of deprecated Python wrapper)

[hunhoffe]

Interesting to hear about the lifetime/ownership issues for pyxrt! Is that for input/output buffers, or just the buffer used internally for the instruction bo, etc. associated with the hardware context?

[thomthehound]

Interesting to hear about the lifetime/ownership issues for pyxrt! Is that for input/output buffers, or just the buffer used internally for the instruction bo, etc. associated with the hardware context?

The funny thing is, I don't see anything in upstream pyxrt.cpp that could even plausibly cause this behavior. In fact, I couldn't find anything anywhere in the publicly available XRT source that explains it. So I'd hesitate to call this a pyxrt issue specifically; I think it may be a coincidence that only the Python binding tests trigger it, because the C++ tests tend to be much more narrowly scoped and do not stress lifetime/ownership/cache behavior (at least not in the way that the bindings tests are designed to do).

My interpretation is that this is probably something specific to the Windows runtime/driver. But the source code for that isn't available to me, so I had to reason about it experimentally. To answer your question, though: pretty much every time, the trigger I observed was eviction/destruction of the cached instruction BO. I did not personally observe problems with the input/output buffers, but I also don't have enough visibility into the closed-source side of Windows to say that could never be an issue, and I didn't try to force it as a negative test, either.

That's a great question, though. I'll try to probe around more regarding this issue while I finalize my hostruntim.py patch, but I'm not sure how much I'll turn up.

[thomthehound]

I don't think I've ever been so excited to see errors on the CI before (and, really, I am truly charmed when it comes to CI luck). Both of these are actually hugely helpful.

@hunhoffe A new wild transient has appeared!

terminate called after throwing an instance of 'xrt_core::system_error'
what(): Open /dev/accel/accel0 failed (err=-19): No such device

I will be modifying run_on_npu.py to be less specific about the wording (it was looking for "No such device with index" before). I'm not sure if there has been an update to the xrt-smi in the CI runner, or if we caught a rare legendary. And, as usual, pretty much no information as to why it happened:

-- Testing: 1276 of 1277 tests, 12 workers --
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60
FAIL: AIE_TEST :: npu-xrt/dynamic_object_fifo/sliding_window/aie2.py (820 of 1276)
******************** TEST 'AIE_TEST :: npu-xrt/dynamic_object_fifo/sliding_window/aie2.py' FAILED ********************
Exit Code: 250

Command Output (stdout):
--


****** Bootgen v2023.2
  **** Build date : May 18 2026-09:48:41
    ** Copyright 1986-2022 Xilinx, Inc. All Rights Reserved.
    ** Copyright 2022-2023 Advanced Micro Devices, Inc. All Rights Reserved.


[INFO]   : Bootimage generated successfully

XRT Build Version: 2.21.0 (HEAD)
       Build Date: 2026-02-18 10:35:42
          Hash ID: 4eb1f4392a012b4e6eca759762389c612537f7c7
Creating a default 'in-memory' xclbin image.

Section: 'MEM_TOPOLOGY'(6) was successfully added.
Size   : 88 bytes
Format : JSON
File   : 'aie2.mlir.prj/main_mem_topology.json'

Section: 'AIE_PARTITION'(32) was successfully added.
Size   : 3608 bytes
Format : JSON
File   : 'aie2.mlir.prj/main_aie_partition.json'
Info: Embedded Metadata section is missing project.platform.device.core element, adding it.
Successfully wrote (9641 bytes) to the output file: final.xclbin
Leaving xclbinutil.
Compilation completed successfully
terminate called after throwing an instance of 'xrt_core::system_error'
  what():  Open /dev/accel/accel0 failed (err=-19): No such device

--
Command Output (stderr):
--
RUN: at line 10: /home/github/actions-runner/_work/mlir-aie/mlir-aie/build/bin/xchesscc_wrapper aie2 -I /opt/ryzen_ai-1.3.0.1/vitis_aie_essentials/include -c /home/github/actions-runner/_work/mlir-aie/mlir-aie/test/npu-xrt/dynamic_object_fifo/sliding_window/kernel.cc -o ./kernel.o
+ /home/github/actions-runner/_work/mlir-aie/mlir-aie/build/bin/xchesscc_wrapper aie2 -I /opt/ryzen_ai-1.3.0.1/vitis_aie_essentials/include -c /home/github/actions-runner/_work/mlir-aie/mlir-aie/test/npu-xrt/dynamic_object_fifo/sliding_window/kernel.cc -o ./kernel.o
RUN: at line 11: "/home/github/actions-runner/_work/mlir-aie/mlir-aie/aie-venv/bin/python" /home/github/actions-runner/_work/mlir-aie/mlir-aie/test/npu-xrt/dynamic_object_fifo/sliding_window/aie2.py > ./aie2.mlir
+ /home/github/actions-runner/_work/mlir-aie/mlir-aie/aie-venv/bin/python /home/github/actions-runner/_work/mlir-aie/mlir-aie/test/npu-xrt/dynamic_object_fifo/sliding_window/aie2.py
RUN: at line 12: "/home/github/actions-runner/_work/mlir-aie/mlir-aie/build/bin/aiecc"  --no-aiesim --aie-generate-npu-insts --aie-generate-xclbin --no-compile-host --xclbin-name=final.xclbin --npu-insts-name=insts.bin ./aie2.mlir
+ /home/github/actions-runner/_work/mlir-aie/mlir-aie/build/bin/aiecc --no-aiesim --aie-generate-npu-insts --aie-generate-xclbin --no-compile-host --xclbin-name=final.xclbin --npu-insts-name=insts.bin ./aie2.mlir
warning: overriding the module target triple with pdarch-unknown-unknown-elf [-Woverride-module]
1 warning generated.
RUN: at line 13: "/usr/bin/clang" /home/github/actions-runner/_work/mlir-aie/mlir-aie/test/npu-xrt/dynamic_object_fifo/sliding_window/test.cpp -o test.exe -std=c++17 -Wall -I/opt/xilinx/xrt/include -L/opt/xilinx/xrt/lib -luuid -lxrt_coreutil -lrt -lstdc++ -I/home/github/actions-runner/_work/mlir-aie/mlir-aie/build/runtime_lib/x86_64/test_lib/include -L/home/github/actions-runner/_work/mlir-aie/mlir-aie/build/runtime_lib/x86_64/test_lib/lib -ltest_utils
+ /usr/bin/clang /home/github/actions-runner/_work/mlir-aie/mlir-aie/test/npu-xrt/dynamic_object_fifo/sliding_window/test.cpp -o test.exe -std=c++17 -Wall -I/opt/xilinx/xrt/include -L/opt/xilinx/xrt/lib -luuid -lxrt_coreutil -lrt -lstdc++ -I/home/github/actions-runner/_work/mlir-aie/mlir-aie/build/runtime_lib/x86_64/test_lib/include -L/home/github/actions-runner/_work/mlir-aie/mlir-aie/build/runtime_lib/x86_64/test_lib/lib -ltest_utils
RUN: at line 14: "/home/github/actions-runner/_work/mlir-aie/mlir-aie/aie-venv/bin/python" "/home/github/actions-runner/_work/mlir-aie/mlir-aie/utils/run_on_npu.py" "npu1" ./test.exe
+ /home/github/actions-runner/_work/mlir-aie/mlir-aie/aie-venv/bin/python /home/github/actions-runner/_work/mlir-aie/mlir-aie/utils/run_on_npu.py npu1 ./test.exe

--

The other useful piece of information gained here is that my float32 error in test_algorithms.py was reproduced on the CI runner. Because of an error in my own code, the CI runner was accidentally forced to run using Peano (and NPU2) and not Chess, but that proves the failure is not specific to Windows. We will not be able to implement fully-Peano CI runners for any NPU2 platform until it is fixed (or gated).

[thomthehound]

• run_on_npu.py now properly detects new transient error language
• Stop having Peano force execution on NPU2 when Chess wants NPU1

[thomthehound]

@erwei-xilinx I'm sorry to bother you, but can you kick this thing back off again for me?

One of these days I need to talk to the management about becoming a collaborator around here...

[thomthehound]

Interesting to hear about the lifetime/ownership issues for pyxrt! Is that for input/output buffers, or just the buffer used internally for the instruction bo, etc. associated with the hardware context?

@hunhoffe I investigated this more thoroughly, and I can say with a very high degree of confidence now that input/output BOs are not affected by this issue; the instruction BOs are the problem (tested with code configured for arbitrary destruction ordering). I was also able to replicate the same errors on Windows using C++ code that forces destruction in the same order that hostruntime.py currently does. Previously, the C++ tests on Windows were shielded from this issue because RAII enforces destruction of the objects in the correct order unless explicitly made not to. At this point, I think the Python tests are passing on Linux because Linux temporarily preserves some handle, resource, or other state after hw_context destruction that Windows does not. I no longer think this is an XRT bug at any level aside from documentation. It looks to be a pure platform issue.

Later tonight Soon I'll be pushing my hostruntime.py update, and I'll include some new lit tests with an XFAIL on Windows around this issue. It should be interesting to see if the C++ version fails on Linux as well, given that I have to force the bad destruction order there.

Edit: I might have to hold off on the lit tests, given I've slightly altered their preferred syntax here and I don't want to have to drift against my own recommendations.

[hunhoffe]

@thomthehound Thanks for digging into this! And for keeping an eye out for legendary device not found errors! Re: float32 errors, we've had some issues with stack overflows when peano uses more stack space than expected, particularly for float32 JIT functions. Estimating/checking the stack size requirements is a bit of a sticky problem (see: #2347) so we've disabled some of the float32 tests for now for that reason. Not sure if that's related to your errors or not.

[thomthehound]

@thomthehound Thanks for digging into this! And for keeping an eye out for legendary device not found errors! Re: float32 errors, we've had some issues with stack overflows when peano uses more stack space than expected, particularly for float32 JIT functions. Estimating/checking the stack size requirements is a bit of a sticky problem (see: #2347) so we've disabled some of the float32 tests for now for that reason. Not sure if that's related to your errors or not.

No problem! I thought it was a great question! And thank you for the reference regarding the float32 bug. From what you just said, if that isn't directly related, it's at least a next-door neighbor. I really appreciate that. I'll look into this more tomorrow.

But, for a first Peano pass, I think this current work is done. I did not intend this to actually fix issues; just to identify them. I'd just like @jgmelber to take a look, because I think this is kind of thing that would interest him.

[thomthehound]

@hunhoffe I understand you are super-busy right now, but would it be a problem to get this merged soon? #3100 will not pass CI without it.

[hunhoffe]

@thomthehound Sorry for the delay! I think either @jackl-xilinx, @erwei-xilinx, or @jgmelber is going to do some windows environment setup trials but I don't think either of them will have time to address this until mid-next week at earliest. I do think this is a good contribution, and I've definitely not forgotten that this is waiting in the queue.

... if you happen to be looking for something to do.... I'm hoping to put out a RFC issue later today about some compile/build/jit/programming example consolidation I'm working on. Here's an exploratory notebook: https://github.com/Xilinx/mlir-aie/blob/ae4cb4e131de1cc9a4d21eca977b5f5c3b011f2b/programming_guide/whats_new_in_unify_compilation_workflow.ipynbf (WIP PR is here: #3025. although I don't plan to merge it in one piece).

I'd be interested to see if you think my proposed changes will play well with Windows. I'll post the official issue with a better writeup soon!

[thomthehound]

@thomthehound Sorry for the delay! I think either @jackl-xilinx, @erwei-xilinx, or @jgmelber is going to do some windows environment setup trials but I don't think either of them will have time to address this until mid-next week at earliest. I do think this is a good contribution, and I've definitely not forgotten that this is waiting in the queue.

... if you happen to be looking for something to do.... I'm hoping to put out a RFC issue later today about some compile/build/jit/programming example consolidation I'm working on. Here's an exploratory notebook: https://github.com/Xilinx/mlir-aie/blob/ae4cb4e131de1cc9a4d21eca977b5f5c3b011f2b/programming_guide/whats_new_in_unify_compilation_workflow.ipynbf (WIP PR is here: #3025. although I don't plan to merge it in one piece).

I'd be interested to see if you think my proposed changes will play well with Windows. I'll post the official issue with a better writeup soon!

No worries! It's just always hard as an outsider to know what's going on. I was just checking in. I appreciate the update!

That work you pointed out looks impressive, exciting and... substantial. I will try to come up with some tests for it against the tree as it stands as well as my backlog of Windows work. From my preliminary look, I don't see anything obviously breaking on the Windows-side, but I did run into some JIT issues while working on my upcoming programming_examples refactor. I think I can probably spin off the institutional fixes for those into a tiny pre-patch.