Frequently Asked Questions

Why create a new test runner?

There are three main reasons: speed, usability and expressibility.

Speed - When you can run all your tests in a matter of seconds, it changes the way you write code. I'm the kind of person who overclocked his CPU when it started taking longer than 3 seconds to run all unit tests in a project. My computer has had 4 or more CPU cores since 2007, but even today in 2013 they are very rarely used for anything. Parallelising test execution would be an excellent raison d'être for all those CPU cores. Also tricks such as class loader caching that are necessary on the JVM to make unit tests more parallelisable, because especially fast unit tests are predominated by class loading overhead.

Usability - It has bothered me for a long time that when you select a test in IntelliJ IDEA's test results (probably also other IDEs), it's not sure that you will see what that test printed. Sometimes the output is shown under a preceding or following test, which makes it hard to debug using print statements. As is often the case, maximizing usability at the user interface level requires changes throughout the whole software stack.

Expressibility - When writing my own testing framework (which uses closures a lot), I faced an impedance mismatch with JUnit. The JUnit test runner has an implicit assumption that it's possible to find out what tests exist before any tests are run. That's reasonable for JUnit, but not all testing frameworks. As a result, those other testing frameworks must rely on horrible hacks, suffer bugs, or be more limited in what they can do.

Why Jumi reports "Pass: 2" even though I ran only one test?

It's because in Jumi the concept of a "test" is slightly different from JUnit and other test runners, due to Jumi's more flexible execution model.

Let's say you have a test class like this:

import org.junit.Test;
public class ExampleTest {
    @Test
    public void testOne() {
    }
}

...and when you run it, the results will look like this:

 > Run #1 in ExampleTest
 > + ExampleTest
 >   + testOne
 >   - testOne
 > - ExampleTest

Pass: 2, Fail: 0

Jumi counts the top-level "ExampleTest" as one test, and the contained "testOne" as one test; in total two tests. This is due to Jumi's flexible execution model which allows arbitrarily nested tests. Think of a "test" as an element in an XML document - they form a tree structure and there is one root. Each test inside a test class is identified by a TestId and the test class itself has the ID TestId.ROOT.