Surefire not reporting on some dynamic tests

[tnleeuw]

I have a situation in my project that kind of looks like this ticket: #384

But is not quite the same.

In the project we have a test-class RunCypressE2eTest that has a @TestFactory annotation and it works as expected. Surefire reports that 7 tests are executed.

I'm trying to add, in the same module of the project, a new test-class RunLarvaTests.

I can see that Surefire picks it up and if I add a test with the @Test annotation (as was my initial setup) then Surefire reports on that test.

However I'm trying to convert it to dynamic tests with a @TestFactory method and I can see that Surefire does run the tests -- but it's not reporting on the dynamic tests, and not failing the build on any of the dynamic tests failing.
However by runtime and output of the test I can see that they have executed.

If I leave in the one test-method with @Test annotation then Surefire will report that 1 test has been executed and if I remove that method, then Surefire will report that 0 tests have been executed for that class.

When running the tests manually in IntelliJ I do get reporting from IntelliJ on each test executed.

I have not yet created a small project to reproduce the issue as I don't yet know what is the circumstance that breaks it -- One test is picked up by Surefire as expected and the other one isn't. Whatever I do I cannot break the working one and so I don't yet know what exactly to put in such a reproduction case.

The project however is open source so the branch on which I'm working can be inspected here, if anyone is curious and willing to look at it.

This is the test that doesn't work as expected:
https://github.com/frankframework/frankframework/blob/feature/LarvaTestFromMaven/test/src/test/java/org/frankframework/runner/RunLarvaTests.java

And this is the test that does:
https://github.com/frankframework/frankframework/blob/feature/LarvaTestFromMaven/test/src/test/java/org/frankframework/runner/RunCypressE2eTest.java

This is output from the Maven build:

[INFO] --- surefire:3.5.3:test (default-test) @ frankframework-test ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.frankframework.runner.IafTestInitializerTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.60 s -- in org.frankframework.runner.IafTestInitializerTest
[INFO] Running org.frankframework.runner.RunCypressE2eTest
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 96.59 s -- in org.frankframework.runner.RunCypressE2eTest
[INFO] Running org.frankframework.runner.RunLarvaTests
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.12 s -- in org.frankframework.runner.RunLarvaTests
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0

The output of RunLarvaTests should have 485 test-cases, and about half of them should fail currently (due to an issue with the environment in which the tests run, not the actual codebase of the project itself 😬 )

[marcphilipp]

There are similar issues reported against Surefire 3.5.3. Does this also happen with earlier versions?

[tnleeuw]

Thanks for the tip. Do you have any specific suggestion for how far back to go with versions of the Surefire plugin, or just to start trying with 3.5.2?

[marcphilipp]

Since the suspected culprit was introduced in 3.5.3, I'd try reverting to 3.5.2.

[tnleeuw]

Yes, it does work as expected with Surefire 3.5.2.

Thanks again for the hint!

Do you think it's worth raising this as a new bug against Surefire, or do the other two issues already cover it sufficiently?

[marcphilipp]

I think raising another issue specific to your use case would make sense. At the very least, I think they should add a test case for your scenario.

[tnleeuw]

Good point. I will. I think I have an idea now how to create a smaller testcase for reproduction.

[tnleeuw]

Unfortunately I did not manage to reproduce my issue in a small standalone testcase yet. :-(