make check output isn't clear if the test was OK

[pps83]

Perhaps, that's not a common scenario, but I tried to run make check. I got tons of output, some of which looked like errors, and the biggest issue is that it's not even clear if the test failed or not.

Looks like I have tons of errors running make check (I run it with msys2 on win11), that's themake check output I get: make-check.txt

Is it so broken and full of errors because of windows, or it's expected to be this way? It does look like very useless idea to even try to run make check or make test if the output is the same useless on other OSes.

[Cyan4973]

There are many tests, and some of them are error scenarios. Therefore the test log can contain error messages. But it's not necessarily a "problem", it actually proves that the tested binary has detected the error condition as it's supposed to.

The purpose of these test scripts is to be employed in automated test environments.
They terminate with an error signal easy to interpret by scripts, as far as shell commands are concerned.
On a posix system, like msys2, it can be consulted with command echo $?.
It will return 0 in case of success, and any other value is interpreted as error.
Everything else, displayed on console, is just a log, which is essentially never interpreted. Except if there is an error. At which point only, these logs becomes useful for investigation.

It seems your demand is that, on top of the return code, you would like to see a final human readable statement, so that a human possibly unaware of posix conventions can more easily understand if the test suite was successful or not.

[pps83]

If you take a look at the log, it doesn't seem to be all OK.

For example line 1959:

./playTests.sh: line 1693: [: : integer expression expected

Perhaps, that huge unreadable output resulted in some issues being unnoticed (if the issue above is actually an issue).
If would be better to redirect output to some log, while printing test names that are being run with some summary at the end.
I wanted to make a PR of to enable x64 asm for windows builds, but wanted to make sure it's OK by running tests, but these tests made me quite confused.

[Cyan4973]

If any test fails, anywhere in the list,
the expectation is that make check will end immediately with lines like:

make[1]: *** [test-zstd] Error 1
make: *** [check] Error 2

which then can be confirmed with :

echo $?
2

If you haven't seen that, and can confirm echo $? 0, then tests are successful.

I've looked at your provided make-check.txt file, and it seems to conclude that all tests are completed fine.

The reported error ./playTests.sh: line 1693 seems to be a test script issue.
Essentially, this line: if [ "$ZSTD_LIB_EXCLUDE_COMPRESSORS_DFAST_AND_UP" -ne "1" ]; then
is not interpreted cleanly (I suspect that $ZSTD_LIB_EXCLUDE_COMPRESSORS_DFAST_AND_UP isn't defined).
As a consequence the test always fails (which seems to be the correct outcome), and all the tests within the following paragraph are skipped.
These tests were added in #3623 .
While I'm not completely sure what they are checking, I suspect they are meant to be skipped in the "normal" case.

If would be better to redirect output to some log,

The test script is designed to be run this way (or similar) :
make check > log 2>&1
In which case, nothing will be displayed on console (everything will be logged into log),
and only the final error code matters (which can be exploited by a sh test condition).

Once again, the log only matters for investigation, once it's established (with the final error code) that there is a problem.

[pps83]

zlib-ng also uses similar style tests (dir of tests where each compiles to standalone executable). I think zlib-ng uses something that comes with cmake and, as a result, on each platform output is consistent and clean:

Build started at 05:58...
1>------ Build started: Project: RUN_TESTS, Configuration: RelWithDebInfo x64 ------
1>1>
1>Test project D:/work-pps/backtest-engine-modules/zlib-ng-pps/build.cmake
1>      Start  1: example
1> 1/70 Test  #1: example ..........................   Passed    0.01 sec
1>      Start  2: infcover
1> 2/70 Test  #2: infcover .........................   Passed    0.01 sec
1>      Start  3: CVE-2002-0059
1> 3/70 Test  #3: CVE-2002-0059 ....................   Passed    0.03 sec
1>      Start  4: CVE-2004-0797
1> 4/70 Test  #4: CVE-2004-0797 ....................   Passed    0.02 sec
1>      Start  5: CVE-2005-1849

[Cyan4973]

These tests (most particularly make check) are designed to work in minimalistic environments (only sh is needed), without requiring any major dependency. This is a property that is best preserved.

[pps83]

only sh is needed

I don't like to use cmake personally, but it feels like requiring cmake would be less intrusive given that sh doesn't exist on windows.

[Cyan4973]

Partially answered in #4247