Difference between @Isolated and @ResourceLock?

[efenderbosch-atg]

We have some tests that capture logs and since the logging framework is basically a shared resource, there were issues when enabling parallel tests.

I tried putting @ResourceLock('logging') on each test method that called the captureLogs helper method, but that didn't work. Then I tried it on each Specification that uses the captureLogs methods, but that didn't work either. Finally I tried @Isolated and that worked, but I'm confused why @ResourceLock wasn't enough.

[Vampire]

Hard to say without knowing your project. But I guess your problem is, that you get a read-write lock (the default) for all tests that try to get the logs, but you still run other test in parallel that produce (and so pollute) log messages. So to get it correct with @ResourceLock, you probably need to get a read lock on all tests that write log messages and a read-write lock where you actually test the log messages, even if semantics of naming are a bit orthogonal in this specific case.

By using @Isolated it runs without any other test running, so no other test can interfere. You can imagine it as getting a read-write lock for a resource all other tests have a read-lock on.

Alternatively, you could investigate whether there is some way to have dedicated logging contexts for each test (or at least the ones that test logging), so that the tests do not influence each other and can even run in parallel, but that is probably more a question for the logging framework you are using and might also depend on whether the SUT logs multi-threaded or not.

[efenderbosch-atg]

I basically had come to the same conclusion a few hours after I asked. Other tests running in parallel are also logging.

[efenderbosch-atg]

I'm not sure this will help anyone else, but we've had some issues w/ devs forgetting to add the @Isolated annotation to tests that capture logs, so I revisited this problem.

I ended moving the log capturing code to an interface LogCapturingSpecification with default methods, then specs can just implement that interface:

interface LogCapturingSpecification {

    default <R> ResultAndEvents<R> captureLogs(Level level, Class<?> logClass, Closure<R> closure) {
        try (TestAppender testAppender = new TestAppender(level, logClass.name)) {
            R result = closure()
            new ResultAndEvents(result, testAppender.events)
        }
    }

    // etc.

}

And I created a global extension to automatically add the logging resource locks:

class LoggingLockGlobalExtension implements IGlobalExtension {

    private static final String LOGGING = "logging"
    private static final ExclusiveResource READ_LOGGING_EXCLUSIVE_RESOURCE =
        new ExclusiveResource(LOGGING, ResourceAccessMode.READ)
    private static final ExclusiveResource READ_WRITE_LOGGING_EXCLUSIVE_RESOURCE =
        new ExclusiveResource(LOGGING, ResourceAccessMode.READ_WRITE)

    @Override
    void visitSpec(SpecInfo spec) {
        // Skip specs marked as isolated as that is more restrictive than a read-write lock
        if (spec.isAnnotationPresent(Isolated)) {
            return
        }

        boolean implementsLogCapturing = LogCapturingSpecification.isAssignableFrom(spec.reflection)
        if (implementsLogCapturing) {
            // For specs that do capture logs, add READ_WRITE locks at the feature level
            // This ensures exclusive access when a test is manipulating the logging system
            spec.features.each { it -> it.addExclusiveResource(READ_WRITE_LOGGING_EXCLUSIVE_RESOURCE) }
        } else {
            // If the spec doesn't capture logs, add a READ lock at the spec level
            // This allows multiple non-capturing tests to run in parallel
            spec.exclusiveResources.add(READ_LOGGING_EXCLUSIVE_RESOURCE)
        }
    }

}

So now a read-write logging lock is automatically applied to every feature in a spec that implements LogCapturingSpecification and every other spec gets a read logging lock. No more manual annotations needed and test time is barely any different.