Allow override of test report status when test execution fails #652
Description
Describe
JUnit has a well known issue that when test execution fails, it cannot list the failing tests.
- "Test suite failed to run" does not report as a test failure in the xml output · Issue #46 · jest-community/jest-junit
- Include tests loading failures in report · Issue #26 · jest-community/jest-junit
- Proposal: classify uncaught errors in tests as "errors" not as "failures" · Issue #5958 · jestjs/jest - stalled
This means the fail-on-error flag is not effective in breaking the build in this situation. We have worked around this by capturing the exit code and exiting after the test reporter has run (in addition to setting the fail-on-error flag).
However, the report in GitHub still shows a green tick and all the remaining tests as passing and no indication there was a problem during execution.
Proposed solution
Support a force-fail: true input along with a force-fail-description: <some markdown/text> that would be included in the report as an error at the top. This would cause a red cross for the unit test report.
Alternatives considered
Pre-processing the JUnit.xml file to include the information we want and forcing the top level test-suite to show as failed.