Running multiple Karate suites with different configurations in parallel via JUnit 5 @TestFactory

[aadam19]

Hi,

I’m evaluating Karate v2 for API contract testing and have a question about running the same test suite with different configurations in parallel.

Context

We have:

• Multiple test account types, for example different user profiles and markets
• A one-time setup per account type: provision account, authenticate, and get session token
• The same .feature files running against each account type
• Account configurations stored in a JSON file

Proposed approach

The reason I am considering this approach is that I want to load the account configurations dynamically from accounts.json, instead of hardcoding account types in the runner or creating one runner class per configuration. This would make it easier to add, remove, or update account presets without changing Java code.

class KarateRunner {

    record Account(String name, String preset, String market) {}

    @TestFactory
    Stream<DynamicTest> runAll() throws Exception {
        Account[] accounts = new ObjectMapper()
            .readValue(getClass().getResourceAsStream("/accounts.json"), Account[].class);

        return Stream.of(accounts).map(a ->
            DynamicTest.dynamicTest(a.name(), () ->
                Karate.run("classpath:api-tests")
                    .systemProperty("preset", a.preset())
                    .systemProperty("market", a.market())
                    .parallel(5)
            )
        );
    }
}

In karate-config.js, I use karate.callSingle() to provision the account once per suite using the system properties:

function fn() {
  var preset = karate.properties['preset'];
  var region = karate.properties['market'];

  var account = karate.callSingle('classpath:auth/setup.feature', {
    preset: preset,
    region: market
  });

  return {
    baseUrl: account.baseUrl,
    token: account.token,
    userId: account.userId
  };
}

Questions

1. Isolation
   Are karate-config.js variables and callSingle results fully isolated between separate Karate.run() invocations?
   For example, if two DynamicTest entries run concurrently, will each get its own callSingle cache and config variables?

2. JUnit 5 parallelism
   If I enable JUnit 5 parallel execution:

   junit.jupiter.execution.parallel.enabled=true

   will the two DynamicTest entries, each calling Karate.run().parallel(5), run concurrently without interfering with each other?

3. Recommended pattern
   Is @TestFactory the right approach for this use case, or is there a more idiomatic Karate pattern for running the same features against multiple configurations in parallel?

Thanks in advance.

[ptrthomas]

This doesn't sound like a legit "environment switch" requirement, but more of a data-driven testing need. Have you looked at the @setup tag? https://docs.karatelabs.io/reusability/dynamic-scenarios/#setup-tag-for-data-generation - it natively supports JSON or even CSV files, and is parallel-ready

[aadam19]

Hi,
I forgot to mention thet accounts need to be previsioned through an extrrnal service. They're not just plain JSON.

The flow wants us to inject the account "type"/"configuration" (found in JSON) into the external API call to the service, which will return the actual account.

Would @setup still be a valid option?

[ptrthomas]

A quick follow-up — we landed a reference test and a docs section showing exactly this shape (per-account provisioning via an external service, then running the same feature files against each account, in parallel):

• Docs: Running Against Multiple Configurations (new sub-section on the Dynamic Scenarios page).
• Reference test: karate-core/.../parallel/multi-account/ — driver.feature is the entry point; the mock echoes the bearer token back so any token bleed between scenarios would fail the test.
• Commit: d7dc382

On the @TestFactory route: in my experience trying to drive parallelization from JUnit has never really worked cleanly — between JVM-global system properties, separate callSingle caches per Suite, and N separate reports to reconcile, it tends to fight the way Karate already parallelizes scenarios internally.

Running a Suite (not just scenarios) in parallel may be possible, but it isn't something we've focused on yet — no one has asked. If you want to explore it after trying the data-driven approach, please go ahead and share what you find — happy to take feedback and adapt.

[aadam19]

Thank you so much!
I'll try your data driven approach first. Later on I'll try further parallelization if I need to, and I'll let you know!