Partial order for test classes/methods?

[toolforger]

I'd like to specify test interdependencies, so that a test will be run only after some other test succeeded.
E.g. in a batch program where each step depends on the output of the previous step, so if an early step fails its tests, the later steps cannot be reasonably run, let alone tested.
Or if a test for command-line option processing in a module integration test fails, it makes no sense to test that the option is correctly interpreted during processing.
Signature processing tests that can't sensibly run if testing the crypto library reveals bugs.

I know it's possible to use mocks to decouple tests that depend on each other, but mocks come with their own set of problems, tend to be overspecific, can be hard and time-consuming to write, and usually become a never-ending story in maintenance.

I know ClassOrderer and MethodOrderer do test sorting, but they impose total orders and hence are overspecific.
These orderers could be used to complete the partial order imposed by interdependencies into a total one though.

I'm not sure what the best way to determine test interdependencies would be.
One could do call chain analysis, or use some library that calculates a dependency graph, but I don't know if these are options in JUnit.
Or maybe a manual specification using an annotation, such as @DependsOnTest(<name of other test that must succeed before running this one>) (maybe the name can be improved).

[sbrannen]

See related issues and discussions:

• Introduce first-class support for scenario tests #48
• Introduce mechanism for inter-test dependencies #607
• Introduce @DependsOn to prioritize test method execution order #1767
• Support for fail-fast in Ordered tests #3099 (comment)

[toolforger]

#48 does share implementation aspects obviously, but it is conceptually unrelated as it's about having a test that consists of a series of subtests. (In fact I believe that #48 might already be covered and could be closed, but I'm not 100% sure; I added a comment there.)

#607 and #1767 seem to be talking about the same idea, yes.
I do not think that fail-fast should be discussed in the same context though; that's a mechanism that can be added independently.

I can't comment on #3099, so I say it here:
I think this is already supported: Just merge the steps into a single test method; this gives you an arrange - act - assert - act - assert sequence.
Or follow one of the techniques I outlined in #49: Define a tag named e.g. Subtest, make the test suite ignore tests with that tag, and write a normal test method that just calls the subtest methods; that way, if a subtest fails, the whole sequence will fail at the first failed assert call but the user can still start the individual subtests interactively.