interpreting the results of failing sanitizers in CI

[Josverl]

@jepler, I hope you can help with the below:

I'm working on #18362 , and I see failures in both of the sanatize_ jobs.

• https://github.com/micropython/micropython/actions/runs/19270914779

for both the failing jobs the Print failures step prints: nothing

So I assume an error of some kind has been found , but I am at a loss of understand what is wrong , let alone on how to fix it.
did something crash ?

[jepler]

The failing step is ci_unix_coverage_run_mpy_merge_tests and it doesn't show its results in a way that is captured by the Print failures step (unfortunately). It passes/fails by comparing the "out-individual" and "out-merged" files using traditional ("ed style") diff, which we're less used to identifying.

(also it's failing during the "regular" coverage build, as well as during the sanitize builds)

I reproduced it locally with the diff command changed to "diff -u" (the unified diff we're more accustomed to) and it showed this in the middle of the output

--- /tmp/tmp.s3MtPD2TpZ/out-individual	2025-11-11 16:28:36.585106677 -0600
+++ /tmp/tmp.s3MtPD2TpZ/out-merged	2025-11-11 16:28:36.693107473 -0600
@@ -591,1455 +591,3 @@
 2
 2
 SKIP
-__imul__
-__imatmul__

In ci_unix_coverage_run_mpy_merge_tests there's a comment that says "None of the tests should SKIP." However, this is not checked.

With your PR branch there appear to be two skipping tests:

$ ./ports/unix/build-coverage/micropython  tests/basics/class_init_subclass.py
SKIP
$ ./ports/unix/build-coverage/micropython  tests/basics/class_metaclass.py
SKIP

I found the following change useful when debugging this:

diff --git a/tools/ci.sh b/tools/ci.sh
index 42d231c458..3624643b47 100755
--- a/tools/ci.sh
+++ b/tools/ci.sh
@@ -690,6 +690,7 @@ function ci_unix_coverage_run_tests {
 }
 
 function ci_unix_coverage_run_mpy_merge_tests {
+    set -x
     mptop=$(pwd)
     outdir=$(mktemp -d)
     allmpy=()
@@ -702,15 +703,25 @@ function ci_unix_coverage_run_mpy_merge_tests {
         outmpy=$outdir/$test.mpy
         $mptop/mpy-cross/build/mpy-cross -o $outmpy $inpy
         (cd $outdir && $mptop/ports/unix/build-coverage/micropython -m $test >> out-individual)
+        if grep '^SKIP$' $outdir/out-individual; then
+            echo 1>&2 "An individual test SKIPped. This test requires all individual tests pass."
+            echo 1>&2 "First skipping test: $inpy"
+            exit 1
+        fi
         allmpy+=($outmpy)
     done
 
+    if grep '^SKIP$' $outdir/out-individual; then
+        echo 1>&2 "An individual test SKIPped. This test requires all individual tests pass."
+        exit 1
+    fi
+
     # Merge all the tests into one .mpy file, and then execute it.
     python3 $mptop/tools/mpy-tool.py --merge -o $outdir/merged.mpy ${allmpy[@]}
     (cd $outdir && $mptop/ports/unix/build-coverage/micropython -m merged > out-merged)
 
     # Make sure the outputs match.
-    diff $outdir/out-individual $outdir/out-merged && /bin/rm -rf $outdir
+    diff -u $outdir/out-individual $outdir/out-merged && /bin/rm -rf $outdir
 }
 
 function ci_unix_coverage_run_native_mpy_tests {

Basically:

• Test after each individual test whether a SKIP line was added. This will detect the first skipping test and exit.
• additionally/alternately, just check once after the loop (but you can't directly tell which test was a skipping test)
• use diff -u which is more likely to be recognized as a list of differences when inspecting the actions build output (still does not become part of the "print failures" step though)

The ci.sh script is very nice to use locally on Linux (please see https://docs.micropython.org/en/latest/develop/gettingstarted.html#using-ci-sh-locally) -- In this case I used the lines

$ ./tools/ci.sh unix_coverage_build unix_coverage_run_mpy_merge_tests

since that's the CI step that was getting stuck.

I hope this info gets you back on track to figure out what's up!

[Josverl]

Thanks for the analysis, much appreciated.

Im familiar with ci.sh, I used it a lot before mpbuild. But bash is not really in my language folder.

Both those SKIPPED tests are only valid when the relevant macro guards are defined, which they are not in the standard build for now.

But at least I have the beginning of the yarn to unravel

[jepler]

Maybe just move the tests out of basics/?

[Josverl]

Doubly-whammy :
that indeed fixed that issue, and also allowed codecov to run - so now I have more things to fix 🙃

• todo: document this somewhere