Surefire reports do not report on errors

[Glodenox]

The surefire report generation currently ignores any compilation errors, which means that when you test with the build parameter that the build should not fail on test failures (common in build pipelines), compilation errors can easily go unnoticed. The expected outcome would be that such a build becomes unstable due to the errors reported in the surefire report.

When a compilation error occurs a file like this is generated as input for the surefire-report.xsl:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites/>

So no real indication of an error can be processed by that XSL file. In theory one could say that this sort of output usually indicates an error and should be reported as an error, as a testsuite without tests is almost certainly a mistake. But I am almost certain that such empty testsuites are out there in the wild and reporting errors on those now might be annoying.

I don't think xspec-maven-plugin can retrieve details on any actual errors, so maybe xspec/xspec has to be adjusted if we want to report on these errors in the surefire reports...

[cmarchand]

I think if a compile error appears in the xspec process, xspec will end prematurly, and so it can not detect compilation error.

@galtm : what do you think about this ? IMO, there are too much launching processes to handle this directly in XSpec. Probably the ant launch script fails on compilation problem, but it's not used from xspec-maven-plugin.

But xspec-maven-plugin starts xspec from java code, and is surely able to detect such a compilation problem, and report it. Or at least, fail with this compilation problem.

@Glodenox : could you provide a complete test case that could be used to work on this ?

[galtm]

@cmarchand : In xspec/xspec, a compilation error would be <xsl:message terminate="yes"> in XSLT. Is there a caller somewhere up the chain that doesn't notice that the process didn't complete successfully or that stops when you think it should push forward with the error information? If so and if it's in xspec/xspec, then reproduction steps would help me look into it.

I don't understand where the file with content <testsuites/> is coming from. Outside the Maven plugin, if I modify an XSpec file to force a compilation error (e.g., remove x:call element from tutorial/escape-for-regex.xspec) and try to run the test, the process stops before generating the *-result.xml and *-junit.xml files.

[cmarchand]

@galtm :

• If I run a xspec file that tests an XSLT file that does not compile, there is no <xsl:message terminate="yes"/> executed. Saxon throws an error before and terminates.

• the file with <testsuites/> is generated by xpec-maven-plugin, and it groups all the results generated by each xspec file run, in a JUnit format. We also generate an index.html with a resume and links to each file.

[galtm]

• If I run a xspec file that tests an XSLT file that does not compile, there is no <xsl:message terminate="yes"/> executed. Saxon throws an error before and terminates.

Maybe I misunderstand the problem that @Glodenox describes. Is it the XSpec file or the underlying XSLT stylesheet that doesn't compile? I'll check back if @Glodenox says more about how to reproduce the problem.

Thanks for the explanation in your 2nd bulleted item.

[Glodenox]

Hey, sorry for the delay! Other priorities at work cropped up, and then I was on holiday for a week.

I've added a unit test to show the behaviour I'm talking about: Glodenox@90111d7 I've also added an invalid file, as it causes a similar issue.

For both XSpec files and XSLT files, the behaviour isn't ideal if the file doesn't parse or terminates during execution without a proper catch. The surefire report is either missing completely (if invalid file), missing any remaining tests after a terminate="yes" or showing an empty list if the failure was at the first test.

Invalid files are already caught in advance in our pipeline, so we didn't run into that issue ourselves. But I think it would still be best if the surefire reports contain any trace of those errors as well.

[cmarchand]

Nice ! Thanks !

I'll have a look at this next week, it's a conference's week now !

[cmarchand]

Just a question. In such a case, does the index.html generated reports the compilation error ?

[Glodenox]

In the index.html, these are usually reported as "missing". It notices that the test exists, but can't find any results.

[cmarchand]

Ok, I've tried to understand what happens here.

Problem is error occurs when either code encounters a <xsl:message terminate="yes"/> or when parsing xspec or XSLT throws an exception.

And I generate the junit and surefire reports in the same transformation, with various TeeDestination.

So, it'll be difficult to use generated files, which are not valid XML, to add some informations in surefire report.

I'm gonna try fo find another way to do it.