Add support for tuning spec path via C API by bangtianliu · Pull Request #184 · iree-org/fusilli

bangtianliu · 2026-02-23T20:13:42Z

--iree-codegen-tuning-spec-path has been exposed through the C-API by IREE PR iree-org/iree#23404. The PR removes the CLI backend requirement for testing and also update readme accordingly.

Assisted-by: Claude

bangtianliu · 2026-02-23T20:53:58Z

I donot think the CI error is caused by changes from this PR.

sjain-stanford · 2026-02-23T22:45:07Z

I do not think the CI error is caused by changes from this PR.

This is true, but your change to remove export FUSILLI_COMPILE_BACKEND_USE_CLI=1 from test_benchmark_runner.sh exposes sanitizer issues that were hidden because of CLI run in a separate process. The benchmark runner now uses the C-API (in-process) backend of the compiler instead. Since the LSAN leak and UBSan violations happen inside the driver process, these are not hidden anymore.

I looked at the violations and they are in external libraries and we might need to add suppressions - RCCL suppression in lsan_suppressions.txt and a UBSAN suppression for upstream IREE (hip_device.c).

sjain-stanford · 2026-02-24T02:40:55Z

Here's the IREE issue and minimal repro for the RCCL leak: iree-org/iree#23561

Please add to the existing build_tools/sanitizers/lsan_suppressions.txt:

# TODO(iree-org/iree#23561): RCCL library leaks during dynamic symbol initialization.
leak:iree_hal_hip_nccl_dynamic_symbols_initialize

With this suppression, ASAN is fixed but also I don't see the UBSAN error anymore (it might have been a cascade effect).

sjain-stanford · 2026-02-24T05:09:27Z

With this suppression, I don't see the UBSAN error anymore - it might have been a cascade effect.

I dug into this and found an interesting anecdote: UBSAN defaults to continuing (not aborting) on violation (unlike ASAN). So unless something else fails, the UBSAN reports are easy to miss - they don't generate non-zero return code and don't break CI. That's what happened above, the ASAN error broke CI but the logs contained UBSAN violations too. We do force it to abort in Fusilli (by setting -fno-sanitize-recover=undefined) but unfortunately IREE's UBSAN setup doesn't do that. There's a way to force it to abort though from Fusilli: Setting UBSAN_OPTIONS=halt_on_error=1. Will send a separate PR for this change, once that lands you can rebase to keep your change well scoped.

sjain-stanford · 2026-02-24T06:34:15Z

#185

@bangtianliu

## Summary @bangtianliu 's recent [PR](#184) switches the benchmark runner to use the C-API compile backend (instead of CLI) - this exposed a few external LSAN and UBSAN [issues](https://github.com/iree-org/fusilli/actions/runs/22331991157/job/64616213364) in IREE that the subprocess CLI invocation was masking. I've reduced them to minimal self contained reproducers and filed issues in IREE for tracking: 1. **Memory leak in HIP driver RCCL initialization** ([iree-org/iree#23561](iree-org/iree#23561)) - ~1216 bytes leaked during `dlopen(librccl.so)` in IREE HIP driver - Occurs on every driver creation 2. **NULL pointer to memcpy** ([iree-org/iree#23562](iree-org/iree#23562)) - Undefined behavior at `hip_device.c:2561` - Triggered during host-to-device buffer transfers with empty binding tables - Violates `memcpy` nonnull attribute contract While looking into the above, I realized an oddity with UBSAN: UBSAN defaults to continuing (not aborting) on violation (unlike ASAN). So unless something else fails, the UBSAN reports are easy to miss - they don't generate non-zero return code and don't break CI. In the case above, ASAN broke CI but the logs contained UBSAN violations too. We do force abort in Fusilli (by setting `-fno-sanitize-recover=undefined`) but unfortunately IREE's UBSAN config doesn't do that. Therefore to force abort on UBSAN issues within IREE, we've got to set `UBSAN_OPTIONS=halt_on_error=1`. This PR configures UBSAN with strict error handling (`halt_on_error=1`) and suppressions to properly handle undefined behavior in IREE's HIP runtime. This enables catching new bugs immediately while suppressing known upstream IREE issues. ## Changes - **Add UBSAN suppressions file** (`build_tools/sanitizers/ubsan_suppressions.txt`) - Suppresses NULL pointer to memcpy bug in IREE HIP device (iree-org/iree#23562) - **Configure UBSAN_OPTIONS across all test types** - Benchmark tests (`benchmarks/CMakeLists.txt`) - Lit tests (`build_tools/cmake/FusilliTestUtils.cmake`) - CTest targets (`build_tools/cmake/FusilliTestUtils.cmake`) - Sets `halt_on_error=1:print_stacktrace=1` to abort on UB violations - **Add LSAN suppression for RCCL leak** (iree-org/iree#23561) - Suppresses memory leak during `iree_hal_hip_nccl_dynamic_symbols_initialize()` - **Update README** with sanitizer configuration documentation - **Add `-fno-sanitize-merge`** flag to preserve source locations in UBSAN reports **Assisted-by:** Claude Opus 4.6 --------- Signed-off-by: Sambhav Jain <sambhav@alumni.stanford.edu> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

kuhar · 2026-02-24T17:11:07Z

+  REQUIRE(std::filesystem::exists(output.path));
+  REQUIRE(std::filesystem::file_size(output.path) > 0);
+
+  std::filesystem::remove_all(input.path.parent_path());


will this be cleaned up if any of the require checks fail above?

Good catch!

I think you're pointing to a design pattern issue used across multiple test files (test_cache.cpp, test_compile_command.cpp, and test_compile_session.cpp).

You are right, if any of the REQUIRE() checks fail, the cleanup code never runs.

I see the manual remove_all(parent_path()) calls remove the entire cache directory, while the CacheFile destructor with remove=true only removes individual files. However, the cleanup still won't execute if a REQUIRE() fails.

I'll address this in a separate PR to keep this one focused on the tuning spec path via C-API, Sound good? Also cc @sjain-stanford here.

Oh, test_cache.cpp already follows the correct pattern (cleanup outside SECTION). I should apply the same pattern to test_compile_session.cpp and test_compile_command.cpp in a separate PR to fix the cleanup issue.

sjain-stanford

LGTM modulo nits. Thanks for re-enabling the C-API path for the benchmark runner - it helped unmask some of the sanitizer issues as well.

Signed-off-by: Bangtian Liu <liubangtian@gmail.com>

Address review feedback: replace auto with explicit types for clarity. Signed-off-by: Bangtian Liu <liubangtian@gmail.com>

Signed-off-by: Bangtian Liu <liubangtian@gmail.com>

Motivated by the issue pointed by reviewer comments: #184 (comment), this PR add `ScopeExit` RAII helper to `tests/utils.h `that removes cache directories via destructor, ensuring cleanup happens even when tests fail. Implementation uses std::function<void()> for simplicity. **Assisted-by:** Claude --------- Signed-off-by: Bangtian Liu <liubangtian@gmail.com>

bangtianliu force-pushed the feature/tuning-spec-capi-support branch from 0bef8b3 to 3ca2722 Compare February 23, 2026 20:14

bangtianliu marked this pull request as draft February 23, 2026 20:14

bangtianliu force-pushed the feature/tuning-spec-capi-support branch 4 times, most recently from 610c96c to cd67720 Compare February 23, 2026 20:25

bangtianliu force-pushed the feature/tuning-spec-capi-support branch from cd67720 to 47ab2f5 Compare February 23, 2026 20:55

bangtianliu requested review from IanWood1, kuhar and sjain-stanford February 23, 2026 20:59

bangtianliu marked this pull request as ready for review February 23, 2026 20:59

kuhar reviewed Feb 23, 2026

View reviewed changes

Comment thread tests/test_compile_session.cpp Outdated

Comment thread tests/test_compile_session.cpp Outdated

bangtianliu force-pushed the feature/tuning-spec-capi-support branch 2 times, most recently from 32fa38a to ec92085 Compare February 24, 2026 01:02

bangtianliu requested a review from kuhar February 24, 2026 01:10

sjain-stanford mentioned this pull request Feb 24, 2026

Add UBSAN suppressions for IREE runtime violations #185

Merged

bangtianliu force-pushed the feature/tuning-spec-capi-support branch from ec92085 to 0bb96ce Compare February 24, 2026 17:02

sjain-stanford reviewed Feb 24, 2026

View reviewed changes

Comment thread tests/test_compile_session.cpp

sjain-stanford reviewed Feb 24, 2026

View reviewed changes

Comment thread tests/test_compile_session.cpp

kuhar reviewed Feb 24, 2026

View reviewed changes

sjain-stanford reviewed Feb 24, 2026

View reviewed changes

Comment thread README.md Outdated

sjain-stanford approved these changes Feb 24, 2026

View reviewed changes

sjain-stanford reviewed Feb 24, 2026

View reviewed changes

Comment thread tests/test_compile_session.cpp Outdated

bangtianliu force-pushed the feature/tuning-spec-capi-support branch from b1e88c0 to ad9d9e6 Compare February 24, 2026 18:34

bangtianliu force-pushed the feature/tuning-spec-capi-support branch 11 times, most recently from 662cb2b to 6886004 Compare February 25, 2026 04:06

sjain-stanford reviewed Feb 25, 2026

View reviewed changes

Comment thread tests/test_compile_session.cpp Outdated

bangtianliu marked this pull request as draft February 25, 2026 05:00

sjain-stanford reviewed Feb 25, 2026

View reviewed changes

Comment thread tests/test_compile_session.cpp Outdated

bangtianliu force-pushed the feature/tuning-spec-capi-support branch from 6886004 to 3a098a2 Compare February 25, 2026 05:47

bangtianliu marked this pull request as ready for review February 25, 2026 05:47

bangtianliu force-pushed the feature/tuning-spec-capi-support branch 2 times, most recently from c52499d to a9f504d Compare February 25, 2026 05:51

bangtianliu added 2 commits February 25, 2026 00:56

Add support for tuning spec path via C API

b365ee2

Signed-off-by: Bangtian Liu <liubangtian@gmail.com>

Use explicit types in tuning spec tests

0ca2020

Address review feedback: replace auto with explicit types for clarity. Signed-off-by: Bangtian Liu <liubangtian@gmail.com>

bangtianliu force-pushed the feature/tuning-spec-capi-support branch from a9f504d to b543494 Compare February 25, 2026 05:56

sjain-stanford reviewed Feb 25, 2026

View reviewed changes

Comment thread tests/test_compile_session.cpp

sjain-stanford reviewed Feb 25, 2026

View reviewed changes

Comment thread tests/test_compile_session.cpp Outdated

sjain-stanford approved these changes Feb 25, 2026

View reviewed changes

address reviewer comments

88f5d10

Signed-off-by: Bangtian Liu <liubangtian@gmail.com>

bangtianliu force-pushed the feature/tuning-spec-capi-support branch from b543494 to 88f5d10 Compare February 25, 2026 06:08

bangtianliu enabled auto-merge (squash) February 25, 2026 06:11

bangtianliu merged commit d1ed0f5 into iree-org:main Feb 25, 2026
12 checks passed

bangtianliu mentioned this pull request Feb 25, 2026

Fix test cleanup to work even when REQUIRE() fails #191

Merged

Uh oh!

Conversation

bangtianliu commented Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bangtianliu commented Feb 23, 2026

Uh oh!

Uh oh!

Uh oh!

sjain-stanford commented Feb 23, 2026

Uh oh!

sjain-stanford commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sjain-stanford commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sjain-stanford commented Feb 24, 2026

Uh oh!

Uh oh!

Uh oh!

kuhar Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

bangtianliu Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

bangtianliu Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sjain-stanford left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bangtianliu commented Feb 23, 2026 •

edited

Loading

sjain-stanford commented Feb 24, 2026 •

edited

Loading

sjain-stanford commented Feb 24, 2026 •

edited

Loading

bangtianliu Feb 24, 2026 •

edited

Loading