[proposal] in-suite orchestration hooks via karate-boot.js

[ptrthomas]

Background

A pattern has come up repeatedly: users want to programmatically orchestrate
test execution — control which scenarios run, in what order, with what retry
behavior, with what shared state — and the current framework doesn't have a
natural home for it. So they reach for whatever's closest and the result is
awkward.

Two recent examples:

• add support for line filters in karate.call #2863 (@jkeys089): grid runs (BrowserStack, Sauce Labs) where a browser
  session takes minutes to start. The desired flow is "discover scenarios →
  run them sequentially in one session → retry inline on comms-error → defer
  the error if retries exhaust → continue". Today this has to be written as
  a Scenario Outline that loops a JS callWithRetry() function over a
  pre-resolved list of scenarios — see the linked PR for the full v1
  pattern that depended on internal APIs.

• Running multiple Karate suites with different configurations in parallel via JUnit 5 @TestFactory #2857 (@aadam19): same .feature files across N account configurations
  from accounts.json. Today this gets written as a JUnit 5 @TestFactory
  that calls Karate.run().parallel(5) once per account.

These look similar but they want different things. This proposal addresses
only the in-suite case (jkeys089). The cross-suite case (aadam19) is
called out as future work below.

Why feature-files are the wrong home for in-suite orchestration

The current workaround — defining orchestration inside a .feature file
via a Scenario Outline + karate.call(...) loop — has a fatal structural
problem: the report tree is wrong. Every scenario shows up nested under
one parent scenario's call step, instead of as its own top-level scenario
result. So a run of 200 scenarios across 5 browsers becomes 5 deeply-nested
trees in HTML / JSONL / Cucumber JSON, rather than 1000 properly-attributed
scenario results. Pass/fail counters, embeds, retries — all flattened.

The orchestration logic and the test logic want to live at different layers.

Context: karate-boot.js is brand new

Worth flagging up front, because the mechanism is recent enough that most
users won't have come across it yet: karate-boot.js is the first
JS-native suite-level entry point Karate has ever had. It landed only in
the last few releases of v2 and is currently scoped to plugin configuration
(boot.plugin('agent'), etc.) and a small boot.* helper namespace
(boot.env, boot.sysenv, boot.read, boot.log). It runs once per
Suite, before SUITE_ENTER fires, and can register RunListener-backed
plugins. See docs/DESIGN.md → Plugin + karate-boot.js
for the full surface today.

What this proposal does is extend that nascent JS-at-the-suite-level layer
into a second responsibility: declarative lifecycle hooks. And we
anticipate a third in time — JS-driven production and execution of suites
(the karate-runner.js direction sketched under future work). The
end-state Karate is heading toward: a small, coherent JS surface for
operating on suites that gives non-Java teams the same control Java users
have had via RuntimeHook / Runner.Builder — without forcing them into a
Java toolchain.

Proposed direction — karate-boot.js grows lifecycle hooks

JS-native hooks on the existing Suite lifecycle, so users don't need to
author a Java plugin to interpose on execution:

// karate-boot.js
const state = { driver: null, deferredErrors: [] }

boot.on('scenarioEnter', ({ scenario, tags, browser }) => {
  if (tags.includes('skipWhen=sandboxed') && browser?.sandboxed) {
    return { action: 'skip' }
  }
})

boot.on('scenarioFailure', ({ scenario, error, attempt }) => {
  if (attempt < 3) {
    return { action: 'retry' }   // inline retry — same scenario, same session
  }
  if (attempt >= 3 && state.driver) {
    state.driver.quit()           // force fresh browser
    state.driver = null
  }
  state.deferredErrors.push({ scenario: scenario.name, error: error.message })
  return { action: 'defer' }      // record + continue suite
})

boot.on('suiteExit', ({ summary }) => {
  if (state.deferredErrors.length) {
    boot.log(`${state.deferredErrors.length} scenarios failed after retry`)
    boot.write('target/deferred.json', JSON.stringify(state.deferredErrors))
  }
})

Key properties:

• Parity with Java RuntimeHook. A strategic direction for Karate has
  been giving non-Java teams the same level of control that has historically
  only been available through Java-side hook implementations. JS-native
  lifecycle hooks close that gap directly.
• Shared state lives in the boot-file closure (state above), naturally
  scoped to the Suite. No per-scenario plumbing needed.
• Explicit return contract: {action: 'retry' | 'skip' | 'defer' | 'continue'}. More expressive than the current boolean RunListener.onEvent
  return, and the explicit action keyword makes the intent readable at
  the call site (vs. helper-method side effects like info.retry()).
• Report shape stays correct: each scenario is still its own top-level
  result. Retries become a first-class attempts: [{result, duration, error}] array on the existing scenario, not a new nested call tree.
• No new top-level file — same karate-boot.js that already handles
  plugin config.

What this replaces / makes unnecessary

• Custom Java RuntimeHook implementations for in-suite control flow.
• The JS-loop-as-orchestrator pattern (the callWithRetry JS function in
  add support for line filters in karate.call #2863).
• Most of the resolveScenarios use case in add support for line filters in karate.call #2863 — if you can hook the
  runtime, you don't need to enumerate scenarios up-front. (The narrower
  karate.call('file.feature:N') syntax we just landed for retry-from-report
  flows is still useful, but it stops being a workaround for the missing
  orchestration layer.)

Out of scope / future work

• Cross-suite orchestration (Running multiple Karate suites with different configurations in parallel via JUnit 5 @TestFactory #2857). The natural continuation of the
  JS-at-the-suite-level layer this proposal extends: a separate
  karate-runner.js with a runner.* API that replaces the default
  path-scan-then-parallel loop, letting users iterate Suites over a data
  source. The name deliberately echoes the Java-side Runner.Builder API
  so the mental model carries over. Likely shape:

  // karate-runner.js — future, not part of this proposal
  for (const account of runner.read('accounts.json')) {
    runner.run({
      path: 'classpath:api-tests',
      name: account.name,
      systemProperties: { preset: account.preset, market: account.market },
      parallel: 5
    })
  }

  Linked here for context; this proposal is only the in-suite hook surface.

Open questions

1. Retry semantics + report. Inline retry means the report needs a
   first-class attempts[] field. JSONL schema bump. What's the right
   default for the legacy pass/fail counter — count only the final attempt,
   or count every attempt as a separate event? Instinct: final attempt for
   counters, full history in attempts[].
2. Driver / shared-resource lifecycle. Closure-captured state in the
   boot file is the proposed primitive (see state.driver above). Is that
   enough, or do we need first-class "suite-scoped resource" helpers
   (acquire/release patterns, lazy init)?
3. Hook surface. Minimum: suiteEnter/Exit, featureEnter/Exit,
   scenarioEnter/Exit, scenarioFailure. Should stepEnter/Exit /
   httpEnter/Exit also be exposed, or are those better left to the
   existing RunListener Java-plugin path (too noisy / too fine-grained
   for JS)?
4. Relationship to existing configure beforeScenario / afterScenario
   / onStepFailure keys. Those are scenario-scoped (set inside a
   feature). These new boot-level hooks would be suite-scoped (set once for
   the whole run). Both should coexist — but the precedence order needs to
   be spelled out.

Ask

@jkeys089 — if this direction looks promising, we'd suggest holding off on
#2863 (the cascade fix and call-by-line syntax are already on main). The
discovery/resolver part can come back as a hook-based design instead. We'd
love your feedback on:

• Does the hook API as sketched cover your retry-and-defer flow without the
  resolver?
• Driver / shared-session lifecycle: is closure-captured state in
  karate-boot.js enough, or do you have requirements that need first-class
  support?
• Anything in your v1 pattern that wouldn't translate cleanly?

And @aadam19 — the in-suite hooks aren't your case, but the future
karate-runner.js direction in "Out of scope" above is. Comments on whether
that shape would replace your @TestFactory pattern welcome.

[jkeys089]

@ptrthomas this looks awesome! We'd gladly use this right now, as-proposed.

The only thing that isn't totally clear is how we can run through the suite multiple times (e.g. for each browser). Perhaps a new "suite outline" concept similar to the current scenario outline that allows us to drive the entire suite with multiple iterations, each with their own custom state? I think this might work for the #2857 case as well (if I'm understanding correctly). It could potentially live directly in karate-boot.js with a small config like:

boot.configure({
  // restricts how many suite instances may be run in parallel - should it multiply or divide Runner threads? Multiply seems the simplest.
  threads: 2,
  // runs once for each thread and returns outline data to be run sequentially inside the thread - normal Runner thread behavior still applies within the the suite instance
  setup(threadIndex, totalThreadCount) {
    // custom logic to prepare state for each suite instance
    return [{ ... }]
  }
})

This is obviously a much larger scope, but would perfectly address our use case and vastly improve the reporting experience. We're working around this now by (ab)using a highly customized scenario outline but, as you pointed out, the reporting experience is less than ideal (i.e. all tests under a single scenario per browser and all failures / retries reported together side-by-side). Until we have something like this, I think we will still need a mechanism to collect a list of scenarios at runtime to allow:

1. running through the entire suite multiple times (e.g. per browser)
2. retrying failed tests inline

If this (or similar) is something you're interested in pursuing then I agree that there is no need to support runtime scenario resolution.