I've noticed several times that the meaning of the CI results is confusing for contributors:
This looks like it is saying "all engines passed these tests." What it really means is "all engines installed correctly and executed these tests without crashing, regardless of pass or fail."
Likewise, this looks like it means "ChakraCore failed these tests, all other engines passed." What it really means is "ChakraCore failed to install."
I think we could reduce this confusion by automatically posting individual test results to a table in a PR comment. For example, the table for #4052 might look like this:
| Test name |
engine262 |
ESMeta |
JSC |
SpiderMonkey |
V8 |
V8 --harmony |
XS |
| language/expressions/assignment/target-member-computed-reference.js |
❌ |
✅ |
❌ |
✅ |
❌ |
✅ |
❌ |
| language/expressions/assignment/target-super-computed-reference.js |
✅ |
❌ |
✅ |
❌ |
✅ |
❌ |
✅ |
(pass/fail status is entirely made up and not a comment on any engine)
ci_test.sh already runs only the added and modified tests on each engine, and just discards the results; we could easily consume those results to build the table.
I've noticed several times that the meaning of the CI results is confusing for contributors:
This looks like it is saying "all engines passed these tests." What it really means is "all engines installed correctly and executed these tests without crashing, regardless of pass or fail."
Likewise, this looks like it means "ChakraCore failed these tests, all other engines passed." What it really means is "ChakraCore failed to install."
I think we could reduce this confusion by automatically posting individual test results to a table in a PR comment. For example, the table for #4052 might look like this:
(pass/fail status is entirely made up and not a comment on any engine)
ci_test.shalready runs only the added and modified tests on each engine, and just discards the results; we could easily consume those results to build the table.