Exit status can’t be used to distinguish between tests not passing and other errors

[Rinzwind]

The exit status of ‘smalltalkci’ is 1 when tests don’t pass:

$ smalltalkci --smalltalk Pharo64-stable .smalltalk.ston; echo Status: $?
[…]
  Executed 64 Tests with 1 Failures and 0 Errors in 0.17s.
[…]
Status: 1

But also when there is some other error, for example:

$ smalltalkci --smalltalk Pharo64-stable .smalltalk.ston; echo Status: $?
[…]
Downloading Pharo64-stable image...
[…]
curl: (6) Could not resolve host: get.pharo.org
curl failed to download get.pharo.org/64/stable […]
Status: 1

So the status can’t be used in scripts to distinguish between these cases. It would be useful to have a status that specifically means the tests didn’t pass.

[fniephaus]

Good suggestion! Are you aware of any testing framework encoding failure kinds with exit codes? It'd be nice if we could just follow what others have worked out already 🙂

[Rinzwind]

I’m not aware of any convention for testing frameworks in particular. More generally, one convention is to use 1 through n for warnings, and anything above n for errors, another is to use 1 for an unspecified failure and anything above for specific failures. Following the first, the status of ‘smalltalkci’ could be 1 to warn that the tests did not pass, and following the second, the status could be 2 to specify its failure is due to the tests not passing.

I’m not sure whether it’d be useful for the status to also distinguish between the tests not passing due to failures, errors, unexpected passes or some combination of these, and whether it would make sense to also use a non-zero status to warn that the tests passed but that some of them were skipped or declared as an expected failure. The latter would probably have to be introduced as an option to ‘smalltalkci’ to not break existing uses. The status could be say 6 or greater if there was an error in ‘smalltalkci’ itself, else 5 if some tests had an error, else 4 if some tests failed, else 3 if some unexpectedly passed, else 2 if some were declared as expected failures, else 1 if some were skipped, else 0.

[fniephaus]

Yes, something like this sounds reasonable. Maybe we leave some sort of gap between the codes, so that we can add more in the future, but that's something we can discuss in the actual implementation. I'm afraid I don't have any time to work on it myself, but I can help review if someone is willing to take a look.

Fun fact: AmigaOS uses these four exit code:

OK 0
WARN 5
ERROR 10
FAILURE 20