Cucumber support

[pshevche]

Summary

Currently, Cucumber is adequately supported only with JUnit4 runner. See this discussion: #276 (comment)

These tests document the current behavior of the plugin: #277

In this issue, we should investigate (and add) support for fine-grained retries using Cucumber runners for JUnit platform or TestNG.

### Tasks
- [ ] https://github.com/gradle/test-retry-gradle-plugin/issues/278
- [ ] https://github.com/gradle/test-retry-gradle-plugin/issues/164

[lnhrdt]

@pshevche We’ve been living with the limitations of the Test Retry Gradle Plugin with Cucumber and JUnit 5 for the last few months. Recently, our test suite grew so large that it became painful enough to drive us to find a solution. In an effort to clarify your comment regarding a working Cucumber setup, we conducted a thorough test of various testing styles and libraries.

In conclusion, we are downgrading our Cucumber test suite module from JUnit Jupiter to JUnit 4. We hope our analysis can help others who are navigating the issues with this plugin and Cucumber. We’ve updated our reproduction repository and added a summary in the README to share as a reference.
https://github.com/lnhrdt/gradle-plugin-cucumber-retry-issue

Would you mind taking a look and letting us know if this aligns with your understanding?

[pshevche]

@lnhrdt , your conclusions look valid to me. Thank you for the analysis and documentation of your findings. I am sorry that we did not get to work on this issue so far and you had to downgrade your setup to JUnit 4. Supporting this requires some fundamental changes to how Gradle selects tests to run, and, unfortunately, we are not able to look into it due to other competing priorities.

[lnhrdt]

Hi @pshevche, thanks for taking the time to review our findings and for confirming our analysis. We appreciate the clarity, even if it means compromise for now.

Not to expand the scope too far, but we’ve added another column to the table in our analysis that compares parallelism support for each approach. This factor also played a significant role in our strategy, and we believe it could help others who are navigating how to run Cucumber tests with Gradle.

One more question: Is there an upstream issue in the Gradle GitHub repository or a public discussion thread regarding the required fundamental changes to how Gradle selects tests to run? We’d like to follow any updates or discussions on this topic if possible.

[pshevche]

Although it talks only about parameterized tests, this would be the issue. The challenge is that Gradle needs to support more fine-grained test selectors than it currently does.

[lnhrdt]

@pshevche I noticed that today @mpkorstanje added support for "rerun files" in cucumber/cucumber-jvm#3057. Could this change open a path for the Test Retry Gradle plugin to support Cucumber tests on JUnit Platform, or is the Gradle work in gradle/gradle#19897 still the only blocker for within-task retries?

[mpkorstanje]

@lnhrdt rerun files are a cucumber specific solution. And when used with the JUnit (5) Platform comes with a bunch of jank. I mostly added it to fill a capability gap between Cucumbers JUnit 4 and JUnit 5 runners too smooth out the upgrade path.

I don't think these are relevant in anyway for rerunning tests through the JUnit Platform.

With the current state of support for the JUnit Platform I'd suggest using the JUnit Platform Launcher API to write your own test (re-)runner and hook that execution into your build somewhere.