Support dynamic tests with JUnit5/Spock/TestNG/... ("maven-plugin-testing-harness"-like tests)

[Djaytan]

Hello,

I explored this new suggested solution for testing Maven plugins but then I realized it is unable to answer my (specific?) needs (from what I understood so far).

In fact, what I want is very simple: being able to dynamically generate Maven projects (in my case by providing an instance of MavenProject built from a Model). The main goal sought here is to follow DRY principle by avoiding duplicating pom.xml files again and again with only a very small difference not always easy to spot most of the time. Well, with ITF and maven-invoker-plugin, when the need is to only write few tests then that may be ok... But what about the case where you want to write dozens or maybe even hundreds of tests?

The maven-plugin-testing-harness framework allows me to do that. ITF not (except if I'm wrong?).

Here is an example of (more or less generic) test that I have:

public final class MyMojoTest extends AbstractMojoTestCase {

  private final FileSystem imfs = Jimfs.newFileSystem(Configuration.unix());

  // Not used here, but used for making the assertions in fact
  private final Path rootDir = imfs.getPath(".");

  @Override
  protected void tearDown() throws Exception {
    super.tearDown();
    imfs.close();
  }

  public void test_execute_nominalCase() {
    // Assemble
    TestConfig testConfig = NOMINAL_TEST_CONFIG;
    MavenProject mavenProject = generateMavenProject(testConfig);
    createInputFiles(testConfigs);

    // Act
    executeMyPlugin(mavenProject);

    // Assert
    assertGeneratedOutputFiles(testConfig);
  }

  // And then we can imagine lots of other tests here...

  // ...

  private void executeMyPlugin(@NotNull MavenProject mavenProject) {
    try {
      MyMojo mojo =
          (MyMojo) lookupConfiguredMojo(mavenProject, "my-mojo");
      mojo.injectCustomFileSystem(imfs);
      mojo.execute();
    } catch (Exception e) {
      throw new RuntimeException(e);
    }
  }
}

Note: the logic behind the generateMavenProject() method implementation may be not trivial at all depending on how complex the plugin's configuration model is. However, the base implementation of the method is the following one:

public final class MavenProjectGenerator {

  public static @NotNull MavenProject generateMavenProject(
      @Nullable Collection<TestConfig> configs) {
    Model model = new Model();
    model.setModelVersion("4.0.0");
    model.setGroupId("my.group.id");
    model.setArtifactId("a-test-project");
    model.setVersion("0.0.0-TEST");

    Build build = new Build();
    build.addPlugin(generateMyPluginModel(configs));
    model.setBuild(build);

    var mavenProject = new MavenProject(model);
    LOG.info("Generated POM content:\n{}", convertToXmlContent(mavenProject));

    return mavenProject;
  }

  private static @NotNull Plugin generateMyPluginModel(
      @Nullable Collection<TestConfig> configs) {
    Plugin plugin = new Plugin();
    plugin.setGroupId("my.group.id");
    plugin.setArtifactId("my-test-plugin");
    plugin.setConfiguration(generatePluginConfiguration(configs));
    return plugin;
  }

  // And we build the configuration part of the plugin by relying on the Xpp3Dom class...
}

Based on this example, what I would expect from a modern Maven test framework is mainly just to have the possibility to rely on regular JUnit 5 tests with standard annotations like @Test, @ParameterizedTest and so on.
So this means: no abstract class like AbstractMojoTestCase to extend, just a call like... Let's say randomly MojoTestExecutor#execute(MavenProject, String) and that's all! I really think this is achievable.

So this would lead to the following result:

import org.junit.jupiter.api.AutoClose;
import org.junit.jupiter.api.Test;

// We no longer have to extend/implement any class/interface
final class MyMojoTest {

  /*
   * Since JUnit Jupiter v5.11 we can rely on @AutoClose annotation
   * By default, the #close() method is called
   * See: https://junit.org/junit5/docs/current/user-guide/#writing-tests-built-in-extensions-AutoClose
   */
  @AutoClose private final MojoTestExecutor mojoTestExecutor = new MojoTestExecutor();

  @AutoClose private final FileSystem imfs = Jimfs.newFileSystem(Configuration.unix());

  // Not used here, but used for making the assertions in fact
  private final Path rootDir = imfs.getPath(".");

  // Nothing changed here
  public void test_execute_nominalCase() {
    // Assemble
    TestConfig testConfig = NOMINAL_TEST_CONFIG;
    MavenProject mavenProject = generateMavenProject(testConfig);
    createInputFiles(testConfigs);

    // Act
    executeMyPlugin(mavenProject);

    // Assert
    assertGeneratedOutputFiles(testConfig);
  }

  // A lot more simpler!
  private void executeMyPlugin(@NotNull MavenProject mavenProject) {
    /* 
     * Here the proposed method signature is just a straight to the goal solution
     * for sharing my idea without worrying about the details.
     * The last parameter is supposed to permit injection of custom dependencies
     * in the IoC container such as Guice, Spring, Quarkus, ...
     * Here, I'm injecting an in-memory file system for improving tests performances,
     * avoiding polluting the laptop FS in case of clean up failure
     * and ensuring portability across platforms (Windows, Linux, MacOS, ...)
     */
    mojoTestExecutor.execute(mavenProject, "my-mojo", imfs);
  }
}

But maybe what should be done is simply make some evolutions on the maven-plugin-testing-harness framework itself? But in this case then we should consider the fact ITF is only a better alternative to the maven-invoker-plugin.

What do you think?

[khmarbaise]

There are ways to programmatically generate project setups for itf (https://khmarbaise.github.io/maven-it-extension/itf-documentation/usersguide/usersguide.html#_generated_project_setup)

And here are some examples: https://github.com/khmarbaise/maven-it-extension/blob/master/itf-examples/src/test/java/com/soebes/itf/examples/mps/MavenProjectSourcesMoreComplexIT.java

At the moment ITF does not use a virtual file system (like jimfs etc.) it uses real hard drive... but that might be a good idea to support that....

[Djaytan]

@khmarbaise Ah I see, I wasn't aware of such possibility with ITF. Thanks for sharing!

Unfortunately, there is still the need to rely on the itf-maven-plugin whereas ideally I would prefer to use on the already existing maven-surefire-plugin and maven-failsafe-plugin (which would also grant more flexibilities to consumers because of the standard setup they represent: they can decide to rely on JUnit, Spark, TestNG, something else and more).

By providing a way to use ITF as a library instead of a framework (just calling the library when needed/wanted as I suggested in my initial post) this allows consumers to be as flexible as they want (as long as the exposed methods from ITF are expressive enough of course). If they want to write Spark tests: they can. If like me they want to rely on virtual FS like Jimfs, then they can as long as there is a mechanism to inject objects in the Mojo under test. The laziest approach would be just to provide a way to retrieve the generated Mojo instance so that they can inject the in-memory file system myself through a call to an initially defined setter on the Mojo (even if not ideal). However, since Maven already rely on Eclipse SISU for dependency injection... Then the biggest thing is already there to support what is needed in such scenario.

Personnally, I would love to see ITF exposing some methods for quick, easy and flexible setup without having to worry/depend on an opinionated setup imposed by a framework.

[Djaytan]

Finally I was able to reach what I wanted with the maven-plugin-testing-harness framework.

First of all, I created an abstraction on top of the framework so that I can control when to launch the Mojo:

import java.nio.file.FileSystem;
import org.apache.maven.plugin.testing.AbstractMojoTestCase;
import org.apache.maven.project.MavenProject;
import org.eclipse.sisu.plexus.Hints;
import org.jetbrains.annotations.NotNull;

// No test is ever expected to be written here!
@SuppressWarnings({"NewClassNamingConvention", "java:S2187"})
public final class MojoTestExecutor extends AbstractMojoTestCase implements AutoCloseable {

  public void executeMojo(
      @NotNull MavenProject mavenProject, @NotNull String goal, @NotNull FileSystem imfs) {
    try {
      setUp();
      getContainer().addComponent(mavenProject, MavenProject.class, Hints.DEFAULT_HINT);
      getContainer().addComponent(imfs, FileSystem.class, Hints.DEFAULT_HINT);
      lookupConfiguredMojo(mavenProject, goal).execute();
    } catch (Exception e) {
      throw new RuntimeException(e);
    }
  }

  @Override
  public void close() throws Exception {
    super.tearDown();
  }
}

In fact, only the MojoTestExecutor class now depends on the Maven test framework. Furthermore, since I found a way to inject dependencies in the IoC container (it was not trivial to spot out...) then I was able to make the mojo test execution logic more generic. Typically, I was able to inject the in-memory file system and thus get rid of the setter at Mojo level.

Overall, doing so provide me complete freedom on which test framework to use (JUnit 5, Spock, TestNG, ...) and how to customize my Mojo thanks to dependency injection pattern:

import static my.package.maven.MavenProjectGenerator.generateMavenProject;
import static my.package.maven.TestConfigAssert.assertThat;
import static my.package.maven.TestDataSetPreparer.prepareCachesDirectoriesAndDrlFiles;
import static my.package.maven.config.TestConfig.NOMINAL_TEST_CONFIGS;

import my.package.maven.config.TestConfig;
import com.google.common.jimfs.Configuration;
import com.google.common.jimfs.Jimfs;
import java.nio.file.FileSystem;
import java.nio.file.Path;
import org.jetbrains.annotations.NotNull;
import org.junit.jupiter.api.AutoClose;
import org.junit.jupiter.api.Test;

final class MyMojoTest {

  @AutoClose private final MojoTestExecutor mojoTestExecutor = new MojoTestExecutor();
  @AutoClose private final FileSystem imfs = Jimfs.newFileSystem(Configuration.unix());
  private final Path rootDir = imfs.getPath(".").toAbsolutePath().normalize();

  @Test
  void execute_nominalCase() {
    // Assemble
    var testConfigs = NOMINAL_TEST_CONFIGS;
    prepareCachesDirectoriesAndDrlFiles(rootDir, testConfigs);

    // Act
    executeMyMavenPlugin(testConfigs);

    // Assert
    assertThat(testConfigs).onlyExpectedCachesAreWellGeneratedUnderRootDir(rootDir);
  }

  private void executeMyMavenPlugin(@NotNull Iterable<TestConfig> testConfigs) {
    var mavenProject = generateMavenProject(testConfigs);
    mojoTestExecutor.executeMojo(mavenProject, "cache-generation", imfs);
  }
}

And well, the observed performances are clearly satisfying:

Here typically, tests now rely on JUnit 5 Jupiter instead of JUnit 4. In the end, I have a very great flexibility regarding how I want to write my tests. This is what would see well for ITF: a simple but very flexible interface against which we can dispatch the test execution of a Mojo, including the possibility to inject dependencies.

Do you agree on that?