twister build fails but returns exit code of 0

[cmorganBE]

Zephyr v3.1.0 (although I looked and it doesn't look like this changed in master)

Just noticed that our twister builds were failing, and the logs show how/why they are failing (its my fault and I'll fix it), but the issue is that our build scripts are checking the exit-code and twister is returning 0 even though the builds are failing.

See log output below:

$ ./zephyr/scripts/twister --build-only -p nucleo_h743zi -W -T unitTests
ZEPHYR_BASE unset, using "/home/cmorgan/projects/xxx/zephyr"
Renaming output directory to /home/cmorgan/projects/xxx/twister-out.6
INFO    - Zephyr version: zephyr-v3.1.0
INFO    - JOBS: 8
INFO    - Using 'zephyr' toolchain.
INFO    - Building initial testsuite list...
INFO    - Writing JSON report /home/cmorgan/projects/xxx/twister-out/testplan.json
INFO    - Using 'zephyr' toolchain.
INFO    - 2 test scenarios (2 configurations) selected, 0 configurations discarded due to filters.
INFO    - Adding tasks to the queue...
INFO    - Added initial list of jobs to queue

ERROR   - nucleo_h743zi             missionManagerTest/missionManager.missionWriteString FAILED: Build failure
ERROR   - see: /home/cmorgan/projects/xxx/twister-out/nucleo_h743zi/missionManagerTest/missionManager.missionWriteString/build.log
INFO    - Total complete:    1/   2  50%  skipped:    0, failed:    0
ERROR   - nucleo_h743zi             powerManagerTest/powerManager.powerManagerTests    FAILED: Build failure
ERROR   - see: /home/cmorgan/projects/xxx/twister-out/nucleo_h743zi/powerManagerTest/powerManager.powerManagerTests/build.log
INFO    - Total complete:    2/   2  100%  skipped:    0, failed:    0
INFO    - 0 of 2 test configurations passed (0.00%), 2 failed, 0 skipped with 0 warnings in 42.75 seconds
INFO    - 0 test configurations executed on platforms, 2 test configurations were only built.
INFO    - Saving reports...
INFO    - Writing JSON report /home/cmorgan/projects/xxx/twister-out/twister.json
INFO    - Using 'zephyr' toolchain.
INFO    - Writing xunit report /home/cmorgan/projects/xxx/twister-out/twister.xml...
INFO    - Writing xunit report /home/cmorgan/projects/xxx/twister-out/twister_report.xml...
INFO    - Run completed


cmorgan@cmorgan-bec:~/projects/xxx$ echo $?
0

[cmorganBE]

@stephanosio is this a discussion topic? I would have thought that this is a bug in the twister script (returning success exit code when it fails)?

[stephanosio]

I am unable to reproduce this.

When filing a bug report, please use the provided template and fill out all the necessary information including the steps to reproduce.

[cmorganBE]

If I can show how it fails then it can be moved back into being a bug?

Steps to reproduce:

• clone zephyr
• cd tests/kernel/fifo
• edit fifo_api/src/main.c (add gibberish to the file)
• ../../../scripts/twister --build-only -p nucleo_h743zi -W -T fifo_api
• Build will show as failed
• echo $? will print out '0' indicating the script was successful.

$ git diff
diff --git a/tests/kernel/fifo/fifo_api/src/main.c b/tests/kernel/fifo/fifo_api/src/main.c
index 1ccb3c244a..074e98a11a 100644
--- a/tests/kernel/fifo/fifo_api/src/main.c
+++ b/tests/kernel/fifo/fifo_api/src/main.c
@@ -20,6 +20,7 @@
  * @}
  */
 
+asdfsdsasdfas
 #include <ztest.h>
 extern void test_fifo_thread2thread(void);
 extern void test_fifo_thread2isr(void);

../../../scripts/twister --build-only -p nucleo_h743zi -W -T fifo_api
ZEPHYR_BASE unset, using "/home/cmorgan/projects/sondebase/zephyr"
Renaming output directory to /home/cmorgan/projects/sondebase/zephyr/tests/kernel/fifo/twister-out.4
INFO    - Zephyr version: zephyr-v3.1.0
INFO    - JOBS: 8
INFO    - Using 'zephyr' toolchain.
INFO    - Building initial testsuite list...
INFO    - Writing JSON report /home/cmorgan/projects/sondebase/zephyr/tests/kernel/fifo/twister-out/testplan.json
INFO    - Using 'zephyr' toolchain.
INFO    - 2 test scenarios (2 configurations) selected, 1 configurations discarded due to filters.
INFO    - Adding tasks to the queue...
INFO    - Added initial list of jobs to queue

ERROR   - nucleo_h743zi             tests/kernel/fifo/fifo_api/kernel.fifo             FAILED: Build failure
ERROR   - see: /home/cmorgan/projects/sondebase/zephyr/tests/kernel/fifo/twister-out/nucleo_h743zi/tests/kernel/fifo/fifo_api/kernel.fifo/build.log
INFO    - Total complete:    2/   2  100%  skipped:    1, failed:    0
INFO    - 0 of 2 test configurations passed (0.00%), 1 failed, 1 skipped with 0 warnings in 7.54 seconds
INFO    - 0 test configurations executed on platforms, 1 test configurations were only built.
INFO    - Saving reports...
INFO    - Writing JSON report /home/cmorgan/projects/sondebase/zephyr/tests/kernel/fifo/twister-out/twister.json
INFO    - Using 'zephyr' toolchain.
INFO    - Writing xunit report /home/cmorgan/projects/sondebase/zephyr/tests/kernel/fifo/twister-out/twister.xml...
INFO    - Writing xunit report /home/cmorgan/projects/sondebase/zephyr/tests/kernel/fifo/twister-out/twister_report.xml...
INFO    - Run completed
cmorgan@cmorgan-bec:~/projects/sondebase/zephyr/tests/kernel/fifo$ echo $?
0

[cmorganBE]

@stephanosio reproduction case above that uses only zephyr code

[stephanosio]

Yes, but please use the provided template.

[cmorganBE]

@stephanosio opened #52913

[cmorganBE]

Closing this out as we upgraded to zephyr 3.2 and its correctly exiting with a zero value. Not sure how this changed since I reviewed the code to find this code path and didn't see any changes but clearly I missed something.