agentErrorRecoveryBenchmark implemented

Author: AdamBien

README.md

@@ -511,7 +511,7 @@ For a focused recording, pass a custom `.jfc` file enabling only the `airhacks.z
 
 ## Benchmarks
 
-The [`benchmarks/`](benchmarks/) directory holds executable agent benchmarks that score tool-calling behavior against seeded ground truth — no LLM judge — along orthogonal axes whose results can disagree. [`agentLoopBenchmark`](benchmarks/agentLoopBenchmark) drives an agent through a *pointer-chasing* chain (serial loop-following); [`agentParallelismBenchmark`](benchmarks/agentParallelismBenchmark) gives the agent independent lookups and measures whether it *batches* them into one turn or serializes them — the inverse axis. Every run prints one normalized markdown table row (`Benchmark | Model | Size | Calls | Turns | Result`) that pastes directly into the results table in [`benchmarks/README.md`](benchmarks/README.md) — see there for mechanisms and sweep usage.
+The [`benchmarks/`](benchmarks/) directory holds executable agent benchmarks that score tool-calling behavior against seeded ground truth — no LLM judge — along orthogonal axes whose results can disagree. [`agentLoopBenchmark`](benchmarks/agentLoopBenchmark) drives an agent through a *pointer-chasing* chain (serial loop-following); [`agentParallelismBenchmark`](benchmarks/agentParallelismBenchmark) gives the agent independent lookups and measures whether it *batches* them into one turn or serializes them — the inverse axis; [`agentErrorRecoveryBenchmark`](benchmarks/agentErrorRecoveryBenchmark) injects transient tool failures into the chain and measures whether the agent *retries* or gives up — the robustness axis. Every run prints one normalized markdown table row (`Benchmark | Model | Size | Calls | Turns | Result`) that pastes directly into the results table in [`benchmarks/README.md`](benchmarks/README.md) — see there for mechanisms and sweep usage.
 
 ## Skills
 

benchmarks/README.md

@@ -17,14 +17,17 @@ terminal instead.
 |-----------|-------|------|-------|-------|--------|
 | loop | claude-opus-4-8 | 50 | 50 | – | PASS |
 | parallelism | claude-opus-4-8 | 8 | 8 | 1 | PASS |
+| recovery | claude-opus-4-8 | 3 | 12 | – | PASS |
 
-- **Benchmark** — `loop` (pointer chasing) or `parallelism` (parallel discrimination)
+- **Benchmark** — `loop` (pointer chasing), `parallelism` (parallel discrimination), or
+  `recovery` (transient-fault recovery)
 - **Model** — the model reported by the LLM responses: the one actually served, which can
   differ from the configured one (529 fallback, lightmetal's own config)
-- **Size** — the task size knob: chain `depth` for loop, independent `tasks` for parallelism
+- **Size** — the task size knob: chain `depth` for loop, independent `tasks` for parallelism,
+  injected `faults` for recovery
 - **Calls** — total tool calls the agent issued
-- **Turns** — agent turns that issued tool calls; loop does not track turns (`–`)
-- **Result** — `PASS`/`FAIL`: secret match (loop), all values retrieved (parallelism)
+- **Turns** — agent turns that issued tool calls; the serial benchmarks do not track turns (`–`)
+- **Result** — `PASS`/`FAIL`: secret match (loop, recovery), all values retrieved (parallelism)
 
 A sweep appends ready-to-paste rows:
 
@@ -74,6 +77,42 @@ Compare `Calls` to `Size`: more calls means the agent wandered or retried; fewer
 stopped early (often narrating or hallucinating hops instead of calling the tool). Both are
 loop-following failures. On `FAIL`, the expected/actual secret mismatch is printed to stderr.
 
+## agentErrorRecoveryBenchmark — transient-fault recovery (robustness)
+
+The same pointer chase as the loop benchmark — identical system prompt, identical tool
+surface — but every third hop fails exactly once with
+`ERROR: transient failure for key '…' — call again with the same key.` and succeeds on
+retry. The only recovery cue is that error text: nothing in the prompt or the tool
+description announces that failures can happen, so the benchmark measures whether the agent
+**reads and reacts to tool errors**. Retrying continues the walk; giving up truncates the
+secret; a fabricated fragment cannot match the seeded ground truth. `faults` is the size
+knob, and the chain is `3 × faults` hops long so recovery is demanded repeatedly, spread
+over the whole walk.
+
+```mermaid
+flowchart LR
+    S[start key] --> F[follow_pointer]
+    F -->|fragment + next key| F
+    F -->|ERROR: transient| R[retry same key]
+    R --> F
+    F -->|next = END| A[assemble secret]
+```
+
+```
+./agentErrorRecoveryBenchmark          # default 3 faults (9 hops)
+for f in 2 4 8; do ./agentErrorRecoveryBenchmark $f; done
+```
+
+```
+| recovery | claude-opus-4-8 | 3 | 12 | – | PASS |   # 9 hops + 3 retries — ideal
+| recovery | claude-opus-4-8 | 3 | 5 | – | FAIL |    # gave up at the second fault
+```
+
+Ideal `Calls` = hops + faults: each fault costs exactly one retry. `recovered=X/faults` on
+stderr shows how many injected failures were retried past; `Calls` well below hops + faults
+means the agent abandoned the walk or hallucinated past an error — the mismatched secret
+exposes both.
+
 ## agentParallelismBenchmark — parallel discrimination (independence)
 
 The inverse axis. The agent is given `tasks` **independent** `id → value` pairs with every id

benchmarks/agentErrorRecoveryBenchmark

@@ -0,0 +1,31 @@
+#!/usr/bin/java --enable-native-access=ALL-UNNAMED -Dtools.permissions.default=allow --class-path=../zsmith/zbo/zsmith.jar:../zsmith/zbo/lightmetal.jar --source 25
+
+import airhacks.zsmith.agent.boundary.Agent;
+import airhacks.zsmith.benchmark.boundary.ErrorRecoveryBenchmark;
+import airhacks.zsmith.tools.control.WriteClipboardTool;
+import airhacks.zsmith.tui.control.CommandLine;
+
+void main(String... args) {
+
+    var faults = CommandLine.firstInt(args, 3);
+
+    var benchmark = new ErrorRecoveryBenchmark(faults);
+
+    // The serial walk needs one turn per hop plus one retry turn per injected fault plus one
+    // turn to emit the answer. The extra slack absorbs stray turns — a "thinking" turn with
+    // no tool call, or an extra retry — so an occasional wasted turn does not trip a false FAIL.
+    var maxIterations = benchmark.depth() + faults + 20;
+
+    var agent = new Agent("error-recoverer", benchmark.systemPrompt())
+            .withTool(benchmark.tool())
+            .withMaxIterations(maxIterations);
+
+    var result = benchmark.score(agent.chat("go"));
+
+    // one normalized markdown row on stdout and in the clipboard — ready to paste into the
+    // summary table in README.md; the recovery count goes to stderr so it never ends up in the table
+    var row = result.markdownRow(agent.modelName());
+    IO.println(row);
+    System.err.println(WriteClipboardTool.write(row));
+    System.err.println(result.diagnostics());
+}

zsmith/src/main/java/airhacks/zsmith/benchmark/boundary/ErrorRecoveryBenchmark.java

@@ -0,0 +1,84 @@
+package airhacks.zsmith.benchmark.boundary;
+
+import java.util.LinkedHashSet;
+import java.util.Set;
+
+import airhacks.zsmith.benchmark.control.Chains;
+import airhacks.zsmith.benchmark.control.TransientFaultTool;
+import airhacks.zsmith.benchmark.entity.Chain;
+import airhacks.zsmith.benchmark.entity.Hop;
+import airhacks.zsmith.benchmark.entity.RecoveryResult;
+import airhacks.zsmith.tools.control.ToolHandler;
+
+/**
+ * Error-recovery benchmark for agents. The same pointer chase as
+ * {@link PointerChasingBenchmark} — identical system prompt, identical tool surface — but every
+ * third hop fails exactly once with a transient error and succeeds on retry. The only recovery
+ * cue is the error text, so the benchmark measures whether the agent reads and reacts to tool
+ * errors: retrying continues the walk, giving up truncates the secret, and a fabricated
+ * fragment cannot match the seeded ground truth.
+ *
+ * <p>Compose it into an {@code Agent} and score the reply:
+ * <pre>{@code
+ * var benchmark = new ErrorRecoveryBenchmark(3);
+ * var agent = new Agent("error-recoverer", benchmark.systemPrompt())
+ *         .withTool(benchmark.tool())
+ *         .withMaxIterations(benchmark.depth() + 3 + 20);
+ * var result = benchmark.score(agent.chat("go"));
+ * IO.println(result.markdownRow(agent.modelName()));
+ * }</pre>
+ */
+public class ErrorRecoveryBenchmark {
+
+    static final long DEFAULT_SEED = 0xC0FFEEL;
+    static final int HOPS_PER_FAULT = 3;
+
+    final Chain chain;
+    final TransientFaultTool tool;
+    final int faults;
+
+    public ErrorRecoveryBenchmark(int faults) {
+        this(faults, DEFAULT_SEED);
+    }
+
+    public ErrorRecoveryBenchmark(int faults, long seed) {
+        this.faults = faults;
+        this.chain = Chains.random(faults * HOPS_PER_FAULT, seed);
+        this.tool = new TransientFaultTool(this.chain, faultyKeys(this.chain, faults));
+    }
+
+    /**
+     * Every third hop of the walk fails, spread over the whole chain so recovery is demanded
+     * repeatedly — not just once at a single point.
+     */
+    static Set<String> faultyKeys(Chain chain, int faults) {
+        var keys = new LinkedHashSet<String>();
+        var key = chain.start();
+        for (var index = 0; !Hop.END.equals(key) && keys.size() < faults; index++) {
+            if (index % HOPS_PER_FAULT == 1) {
+                keys.add(key);
+            }
+            key = chain.hop(key).next();
+        }
+        return keys;
+    }
+
+    public ToolHandler tool() {
+        return this.tool;
+    }
+
+    public int depth() {
+        return this.chain.depth();
+    }
+
+    public String systemPrompt() {
+        return PointerChasingBenchmark.chasePrompt(this.chain.start());
+    }
+
+    public RecoveryResult score(String agentAnswer) {
+        var actual = agentAnswer == null ? "" : agentAnswer.replaceAll("\\s", "");
+        var passed = this.chain.secret().equals(actual);
+        return new RecoveryResult(this.faults, this.chain.depth(), this.tool.calls(),
+                this.tool.recovered(), passed, this.chain.secret(), actual);
+    }
+}

zsmith/src/main/java/airhacks/zsmith/benchmark/boundary/PointerChasingBenchmark.java

@@ -47,6 +47,14 @@ public String startKey() {
     }
 
     public String systemPrompt() {
+        return chasePrompt(this.chain.start());
+    }
+
+    /**
+     * Shared with {@link ErrorRecoveryBenchmark}: identical instructions, so recovery behavior
+     * is driven purely by the tool's error text, not by prompt differences.
+     */
+    public static String chasePrompt(String startKey) {
         return """
                You are chasing a chain of pointers to reconstruct a hidden secret.
 
@@ -59,7 +67,7 @@ public String systemPrompt() {
 
                When finished, reply with ONLY the concatenated fragments in the exact
                order you collected them: no spaces, no separators, no extra words.
-               """.formatted(this.chain.start());
+               """.formatted(startKey);
     }
 
     public BenchmarkResult score(String agentAnswer) {

zsmith/src/main/java/airhacks/zsmith/benchmark/control/TransientFaultTool.java

@@ -0,0 +1,71 @@
+package airhacks.zsmith.benchmark.control;
+
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.json.JSONObject;
+
+import airhacks.zsmith.benchmark.entity.Chain;
+import airhacks.zsmith.tools.control.ToolHandler;
+
+/**
+ * Decorates {@link PointerChasingTool} with seeded transient faults: each faulty key fails
+ * exactly once, then succeeds on retry. The recovery cue lives only in the error text — the
+ * tool name, description, and schema are the delegate's, unchanged, so the benchmark measures
+ * the agent's reaction to an error result, not obedience to an upfront warning.
+ */
+public class TransientFaultTool implements ToolHandler {
+
+    static final int FAILURES_PER_KEY = 1;
+
+    final PointerChasingTool delegate;
+    final Set<String> faultyKeys;
+    final Map<String, Integer> attempts;
+    final AtomicInteger calls;
+
+    public TransientFaultTool(Chain chain, Set<String> faultyKeys) {
+        this.delegate = new PointerChasingTool(chain);
+        this.faultyKeys = faultyKeys;
+        this.attempts = new ConcurrentHashMap<>();
+        this.calls = new AtomicInteger();
+    }
+
+    public int calls() {
+        return this.calls.get();
+    }
+
+    /** Faulty keys the agent called again after the failure — the recovery count. */
+    public int recovered() {
+        return (int) this.faultyKeys.stream()
+                .filter(key -> this.attempts.getOrDefault(key, 0) > FAILURES_PER_KEY)
+                .count();
+    }
+
+    @Override
+    public String toolName() {
+        return this.delegate.toolName();
+    }
+
+    @Override
+    public String description() {
+        return this.delegate.description();
+    }
+
+    @Override
+    public JSONObject inputSchema() {
+        return this.delegate.inputSchema();
+    }
+
+    @Override
+    public String execute(JSONObject input) {
+        this.calls.incrementAndGet();
+        var key = input.getString(PointerChasingTool.Field.key.name());
+        var attempt = this.attempts.merge(key, 1, Integer::sum);
+        if (this.faultyKeys.contains(key) && attempt <= FAILURES_PER_KEY) {
+            return "ERROR: transient failure for key '" + key + "' — call again with the same key.";
+        }
+        return this.delegate.execute(input);
+    }
+}

zsmith/src/main/java/airhacks/zsmith/benchmark/entity/RecoveryResult.java

@@ -0,0 +1,22 @@
+package airhacks.zsmith.benchmark.entity;
+
+/**
+ * Outcome of an error-recovery run. {@code recovered} vs {@code faults} shows how many injected
+ * transient failures the agent retried past; {@code passed} is the secret match, which a
+ * fabricated (non-retried) fragment cannot survive. Renders to the shared {@link BenchmarkRow}
+ * columns; recovery detail is exposed via {@link #diagnostics()} for stderr.
+ */
+public record RecoveryResult(int faults, int depth, int calls, int recovered, boolean passed,
+        String expected, String actual) {
+
+    public String markdownRow(String model) {
+        return new BenchmarkRow("recovery", model, this.faults, this.calls, "–", this.passed).markdown();
+    }
+
+    public String diagnostics() {
+        var recovery = "recovered=%d/%d idealCalls=%d".formatted(
+                this.recovered, this.faults, this.depth + this.faults);
+        return this.passed ? recovery
+                : recovery + " expected=%s actual=%s".formatted(this.expected, this.actual);
+    }
+}

zsmith/src/main/java/airhacks/zsmith/benchmark/package-info.java

@@ -28,5 +28,12 @@
  * serializes them. Its tool runs in parallel and gauges its own concurrency, so the metric is
  * efficiency (calls vs turns) rather than a correctness match — a capability that can disagree
  * with loop-following, which is what makes it worth measuring separately.
+ *
+ * <p>{@link airhacks.zsmith.benchmark.boundary.ErrorRecoveryBenchmark} probes the robustness
+ * axis: the identical pointer chase, but seeded hops fail transiently exactly once (see
+ * {@link airhacks.zsmith.benchmark.control.TransientFaultTool}) and succeed on retry. Nothing
+ * in the prompt or tool description announces the faults — the error text is the only cue —
+ * so it measures whether the agent reacts to tool errors by retrying rather than giving up or
+ * fabricating the missing fragment; the seeded secret exposes both failure modes.
  */
 package airhacks.zsmith.benchmark;

zsmith/version.txt

@@ -1 +1 @@
-2026.07.05.02
+2026.07.05.03