Testsuite reports success even when tests crashed

[heplesser]

Under special circumstances, our test setup does not detect crashed tests and reports success where it should not. This problem has occurred during PyNEST-NG testing.

This happens because we use set +e/-e to ensure that the do_tests.sh continues to execute until the end even if a particular set of tests should crash. This is desirable. We then detect possible errors by globbing for *.xml files and analysing their content.

One problematic case is a segfault on test collection (due to #3400), with

• segfault: https://github.com/heplesser/nest-simulator/actions/runs/13130033031/job/36633256531#step:23:213
• success report: https://github.com/heplesser/nest-simulator/actions/runs/13130033031/job/36633256531#step:23:298

In this case, no XML file is generated at all for phase 7, so Phase 7 is simply not reported. As long as no-one looks at the number of tests run, the problem is not detected.

The other problematic case is a failure during testing because of stray arguments in a command (fixed by now in pytest-ng-adac) which looks like this

• failure: https://github.com/heplesser/nest-simulator/actions/runs/13130033031/job/36633256821#step:23:233
• clean report: https://github.com/heplesser/nest-simulator/actions/runs/13130033031/job/36633256821#step:23:281

In this case, XML files are generated, but they just report that there were 0 tests. Note that the error above was triggered by setting the environment variable DO_TESTS_SKIP_TEST_REQUIRING_MANY_CORES=true before invoking make installcheck (it adds added flag that got garbeled due to quote issues, now fixed).

We have the "report only phases with XML files" approach because not every phase is run in every test configuration. But I think what we need to do is that before we start the tests for a given phase, we need to register that that phase is entered and that not finding an XML file for that phase at the end therefore is an error. This will solve the first type of problem.

For the second type of problem, we could consider it an error if the number of tests reported is zero and no other problems (XML parse failure) occured.