Provide a MethodOrderer which enforces unique order values

[OLibutzki]

When a test is annoated with @TestMethodOrder( MethodOrderer.OrderAnnotation.class ) we want the test methods to executed in the order of the @Order annotation.
When twei methods have the same order value the Javadoc states they "will be sortedarbitrarily adjacent to each other". That makes perfect sense.

Imagine this scenario: We have a tets class which contains 6 steps - each of them depends on its predecessor. Each of them is annotated by @Order with an incremented number. Later on we add step between 2 and 3 and accidently copy the @Order annotation from another method.

This might lead to a flaky test as the same order value is used twice.

For sure this is the developer's mistake, but what about defining that each test method has to have an unique order value as we want to control the execution order deterministically?

We could have a special MethodOrderer for this purpose like
@TestMethodOrder( MethodOrderer.EnforcingOrderAnnotation.class )

The whole test class failes if there are at least two methods with the same order value.

[sbrannen]

When twei methods have the same order value the Javadoc states they "will be sortedarbitrarily adjacent to each other".

Although the Javadoc states that (and I may even have written that), I think the resulting order is still deterministic since JUnit guarantees a deterministic ordering for test methods (even if the ordering is "non-obvious"), and since java.util.List.sort() does not change the original order of elements that have the same order value.

Though I might be mistaken about that last point.

[sbrannen]

Some quick research...

• the descriptors are in an ArrayList.
• java.util.ArrayList.sort(Comparator<? super E>) delegates to java.util.Arrays.sort(E[], int, int, Comparator<? super E>).
• The Javadoc for Arrays.sort states: "This sort is guaranteed to be stable: equal elements will not be reordered as a result of the sort."

[OLibutzki]

Thanks for responding that quickly.

I agree that the behaviour seems to be deterministic, so the tests won't be flaky.

Nevertheless I see some value in this issue as you can document the need for an explicit order... as a plus you not only document, but also verify this intention.

One could also think about adding a boolean flag enforceUniqueOrder to @TestMethodOrder.

[OLibutzki]

In my particular case I had two methods which accidently had the same order value.

Took me some time to realize that the test failure was caused by the methods executed in the wrong order.

Having an error message like "You expected the order values to be unique, but method method5 and method6 both have the order value 5." would have helped me.

[OLibutzki]

One last note: I'm quite sure that most of the time it's desired to have an explicit execution order when using @TestMethodOrder.

Falling back to the "JUnit ordering magic" is something you rarely want to have when using @TestMethodOrder.

Well, of course, that's just my personal assumption.

[marcphilipp]

I can see that being useful as well. If we add it, I think there should also be a configuration parameter for changing the default.

[XIANY1203]

I am interested in this issue, may I have a try?