feat: smarter CLI waiting — heartbeat + status file + foreground-wait relay (v0.7.5)

A slow-but-alive CLI backend was mistaken for a failure: the orchestrator
backgrounded run.mjs and gave up on a ~15s blind poll, even though the CLI
answered seconds later within run.mjs's own (generous) timeout. The fix adds
liveness signalling and corrects how the relay waits.

- run.mjs: wrap each backend invoke() with a heartbeat — emit a
  "[mmt] backend '<be>' still running (Ns)…" stderr line every 10s and
  maintain a status file ({state:"running"|"done"|"failed", backend,
  elapsed_ms, …}) at a PREDICTABLE path: "<call-file>.status.json" when
  --call-file is used, else .mmt/calls/<callId>.status.json. stopHeartbeat is
  called on every terminal branch (quota / fail / success). All best-effort:
  the tick is fully try/catch-guarded (incl. an EPIPE-safe stderr write) so it
  can never throw out of the interval or affect the dispatch; the timer is
  unref'd so it never keeps the process alive. Generous timeouts unchanged —
  observability added, not shortened.
- Relay guidance (workflows/team.mjs + reasoning.mjs, commands/team.md +
  reasoning.md, generated agents/*.md): instruct the relay to run run.mjs
  FOREGROUND and WAIT — no backgrounding, no self-imposed sleep/timeout/tail -f,
  re-run and keep waiting on the relay shell's own time limit, and poll the
  status file to confirm liveness. A slow response is explicitly NOT a failure.
- Tests: 93/93 offline (was 91) — a failed-call status:"failed" assertion on
  the existing fake-CLI harness, a --call-file success status:"done" test, and
  foreground-wait + status-file markers in workflows.test.mjs. Docs/badge -> 93.

Codex-verified: live heartbeat fires at 10s on a 12s fake CLI; codex
code-reviewed the diff, flagged one [Medium] (unguarded stderr write in the
tick), which was fixed, and a focused re-review returned PASS.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: WatchAndyTW

.claude-plugin/plugin.json

@@ -2,7 +2,7 @@
   "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json",
   "name": "multi-model-team",
   "displayName": "Multi-Model Team",
-  "version": "0.7.4",
+  "version": "0.7.5",
   "description": "Delegates token-heavy, self-contained tasks to local CLI backends chosen by task size and type, with credit-exhaustion fallback through the backend chain to native Claude and a glanceable statusline HUD. Routing and shipped backend behavior are configured in config/roster.json; adding a new backend kind requires invoker and health support in code. /team runs a decompose → dispatch → verify → fix → synthesize pipeline; an optional config-gated UserPromptSubmit hook can proactively nudge delegation.",
   "author": {
     "name": "WatchAndyTW",

CLAUDE.md

@@ -9,7 +9,7 @@ glanceable statusline HUD.
 (the agy lane runs under a real pseudo-terminal — ConPTY on Windows, forkpty on POSIX); everything
 else is Node stdlib. Cross-platform (Windows/Linux/macOS). `package.json` `"type":"module"`.
 
-**Status:** built, adversarially reviewed, and green. `npm test` passes **91/91** offline
+**Status:** built, adversarially reviewed, and green. `npm test` passes **93/93** offline
 (plus live agy + codex smoke tests under `MMT_LIVE=1`). Two live backends: **agy** (Gemini)
 and **codex** (OpenAI Codex CLI); opencode remains a config-only stub. codex also serves as the
 **`/team` verifier**. See `README.md` (user-facing), `PROBES.md` (grounded CLI findings), and
@@ -335,7 +335,7 @@ fallback hop.
 ## Testing
 
 ```bash
-npm test                         # offline: 91/91 routing + unit tests (no backend calls)
+npm test                         # offline: 93/93 routing + unit tests (no backend calls)
 MMT_LIVE=1 npm test              # + live agy + codex smoke tests (network required)
 ```
 

README.md

@@ -8,7 +8,7 @@ Multi-model orchestration for Claude Code. Route by task, fan out in parallel, f
 
 ![Node](https://img.shields.io/badge/node-%3E%3D18-339933?logo=node.js&logoColor=white)
 ![Type](https://img.shields.io/badge/module-ESM-f7df1e)
-![Tests](https://img.shields.io/badge/tests-91%2F91%20passing-3fb950)
+![Tests](https://img.shields.io/badge/tests-93%2F93%20passing-3fb950)
 ![Platforms](https://img.shields.io/badge/platform-win%20%7C%20linux%20%7C%20macOS-555)
 ![Deps](https://img.shields.io/badge/runtime%20deps-1%20(node--pty)-blue)
 
@@ -276,7 +276,7 @@ docs/REASONING.md            design contract for the /reasoning Fusion pipeline
 ## 🧪 Testing
 
 ```bash
-npm test                # offline: 91/91 routing + unit tests (no backend calls)
+npm test                # offline: 93/93 routing + unit tests (no backend calls)
 MMT_LIVE=1 npm test     # also run live agy + codex smoke tests (network required)
 ```
 

agents/agy.md

@@ -47,6 +47,13 @@ You are the **agy** dispatcher for the multi-model-team plugin. You do **not** s
    - If the task references a local file/dir the backend should read itself, add
      `--add-dir "<dir>"` so the backend reads it on its own quota instead of through Claude.
    - Do NOT inline the task on the command line and do NOT add commentary to the prompt.
+   - **Run it in the FOREGROUND and WAIT.** The agy CLI can take several minutes on a
+     hard task; run.mjs blocks until it finishes (it has its own generous timeout). Do NOT
+     background it (no `&`, no `run_in_background`), do NOT wrap it in your own
+     `sleep`/`timeout`/`tail -f`, and do NOT give up early — a slow response is NOT a failure.
+     If your shell hits its own time limit, run the SAME command again and keep waiting; run.mjs
+     emits a `[mmt] backend still running (Ns)…` heartbeat to stderr and writes a
+     `<call-file>.status.json` ({state:"running"|"done"|"failed"}) you can read to confirm it's alive.
 4. Interpret the output:
    - If stdout begins with `MMT_NATIVE_HANDOFF`, the **agy** CLI was unavailable/exhausted (it fell through the fallback chain) — return that sentinel verbatim so the orchestrator (Opus/Sonnet) handles it in-context.
    - Otherwise stdout **is** the delegated result. Return it **verbatim** — no analysis, no

agents/codex.md

@@ -47,6 +47,13 @@ You are the **codex** dispatcher for the multi-model-team plugin. You do **not**
    - If the task references a local file/dir the backend should read itself, add
      `--add-dir "<dir>"` so the backend reads it on its own quota instead of through Claude.
    - Do NOT inline the task on the command line and do NOT add commentary to the prompt.
+   - **Run it in the FOREGROUND and WAIT.** The codex CLI can take several minutes on a
+     hard task; run.mjs blocks until it finishes (it has its own generous timeout). Do NOT
+     background it (no `&`, no `run_in_background`), do NOT wrap it in your own
+     `sleep`/`timeout`/`tail -f`, and do NOT give up early — a slow response is NOT a failure.
+     If your shell hits its own time limit, run the SAME command again and keep waiting; run.mjs
+     emits a `[mmt] backend still running (Ns)…` heartbeat to stderr and writes a
+     `<call-file>.status.json` ({state:"running"|"done"|"failed"}) you can read to confirm it's alive.
 4. Interpret the output:
    - If stdout begins with `MMT_NATIVE_HANDOFF`, the **codex** CLI was unavailable/exhausted (it fell through the fallback chain) — return that sentinel verbatim so the orchestrator (Opus/Sonnet) handles it in-context.
    - Otherwise stdout **is** the delegated result. Return it **verbatim** — no analysis, no

commands/reasoning.md

@@ -153,6 +153,14 @@ with no preamble:
 
 node "<PLUGIN_ROOT>/src/bin/run.mjs" --call-file="<CALL_PATH>"
 
+CRITICAL — run it in the FOREGROUND and WAIT for it to finish. The <BE> CLI can legitimately take
+several minutes; run.mjs blocks until it completes (it has its own generous timeout). Do NOT
+background it (no `&`, no run_in_background), do NOT wrap it in your own sleep/timeout/tail -f, and do
+NOT give up early — a slow response is NOT a failure. If your Bash tool hits its own time limit, run
+the SAME command again and keep waiting; run.mjs prints a "[mmt] backend still running (Ns)…"
+heartbeat to stderr and writes "<CALL_PATH>.status.json" ({state:"running"|"done"|"failed"}) you can
+read to confirm it is still alive.
+
 If stdout begins with "MMT_NATIVE_HANDOFF" (the <BE> CLI was unavailable), return EXACTLY that
 sentinel line and nothing else — do not answer the question yourself.
 ```

commands/team.md

@@ -150,6 +150,14 @@ alone. There are two worker kinds:
 
   node "<PLUGIN_ROOT>/src/bin/run.mjs" --call-file="<CALL_PATH>"
 
+  CRITICAL — run it in the FOREGROUND and WAIT for it to finish. The <BE> CLI can legitimately take
+  several minutes on a hard task; run.mjs blocks until it completes (it has its own generous timeout).
+  Do NOT background it (no `&`, no run_in_background), do NOT wrap it in your own sleep/timeout/tail -f,
+  and do NOT give up early — a slow response is NOT a failure. If your Bash tool hits its own time
+  limit, run the SAME command again and keep waiting; run.mjs prints a "[mmt] backend still running
+  (Ns)…" heartbeat to stderr and writes "<CALL_PATH>.status.json" ({state:"running"|"done"|"failed"})
+  you can read to confirm it is still alive.
+
   If stdout begins with "MMT_NATIVE_HANDOFF" (the <BE> CLI was unavailable), return EXACTLY that
   sentinel line and nothing else — do not solve the task yourself. Otherwise return stdout as printed.
   ````

package-lock.json

@@ -1,6 +1,6 @@
 {
   "name": "multi-model-team",
-  "version": "0.7.4",
+  "version": "0.7.5",
   "type": "module",
   "private": true,
   "description": "Delegates token-heavy, self-contained tasks to local CLI backends (agy/Gemini, codex) chosen by task size and type, with credit-exhaustion fallback to native Claude and a statusline HUD. Node ESM, zero-build, cross-platform (Windows/Linux/macOS).",

package.json

@@ -118,6 +118,56 @@ function failuresLogPath() {
   return path.join(base, 'failures.log');
 }
 
+// --- progress heartbeat + status file ---------------------------------------
+// A CLI backend can legitimately take minutes (agy/codex on a hard task). The OLD behaviour gave the
+// orchestrator no signal that a slow call was still ALIVE — so a Claude-side poll that timed out at
+// ~15s looked like a failure even though the CLI answered seconds later. While a call is in flight
+// run.mjs now (a) emits a `[mmt] …still running (Ns)` heartbeat to stderr every HEARTBEAT_MS, and
+// (b) maintains a status file at .mmt/calls/<callId>.status.json the orchestrator can poll —
+// {state:"running"|"done"|"failed", backend, elapsed_ms, ...}. This makes "slow but alive"
+// distinguishable from "dead" without shortening the (generous) hard_timeout. All best-effort: a
+// status/heartbeat write that fails NEVER affects the actual dispatch.
+const HEARTBEAT_MS = 10_000;
+
+// Resolve the status-file path. PREDICTABLE so the orchestrator can poll it: when a --call-file was
+// used, it is "<call-file>.status.json" (sits right next to the payload the relay wrote). Otherwise
+// it falls back to .mmt/calls/<callId>.status.json (or MMT_LOG_DIR's parent /calls when set).
+function statusFilePath(callId, callFile) {
+  if (callFile) return `${callFile}.status.json`;
+  const base = process.env.MMT_LOG_DIR ? path.dirname(process.env.MMT_LOG_DIR) : path.join(process.cwd(), '.mmt');
+  return path.join(base, 'calls', `${callId}.status.json`);
+}
+
+function writeStatus(statusFile, callId, obj) {
+  if (!statusFile) return;
+  try {
+    fs.mkdirSync(path.dirname(statusFile), { recursive: true });
+    fs.writeFileSync(statusFile, JSON.stringify({ callId, updated: new Date().toISOString(), ...obj }) + '\n', 'utf8');
+  } catch { /* status is observability-only — never break the run */ }
+}
+
+// Start a heartbeat for a backend call. Returns a stop(finalObj) that clears the timer and writes the
+// terminal status. Writes an initial "running" status immediately so a poll right after spawn sees it.
+function startHeartbeat({ callId, statusFile, backend: be, model, tier, rule, startMs }) {
+  writeStatus(statusFile, callId, { state: 'running', backend: be, model, tier, rule, elapsed_ms: 0 });
+  const tick = () => {
+    // Best-effort: a heartbeat must NEVER throw out of the interval callback (an uncaught throw there
+    // is an unhandledException). Guard the stderr write (EPIPE/closed stream) and the status write.
+    try {
+      const elapsed = Date.now() - startMs;
+      const secs = Math.round(elapsed / 1000);
+      try { process.stderr.write(`[mmt] backend '${be}' still running (${secs}s)… [call=${callId}]\n`); } catch { /* stderr closed/EPIPE */ }
+      writeStatus(statusFile, callId, { state: 'running', backend: be, model, tier, rule, elapsed_ms: elapsed });
+    } catch { /* heartbeat is observability-only — never affect the dispatch */ }
+  };
+  const timer = setInterval(tick, HEARTBEAT_MS);
+  if (typeof timer.unref === 'function') timer.unref();
+  return function stop(finalObj) {
+    clearInterval(timer);
+    writeStatus(statusFile, callId, { backend: be, model, tier, rule, ...finalObj });
+  };
+}
+
 function logFailure({ backend: be, model, tier, rule, code, durMs, stderr, kind, callId }) {
   // 1. Loud, human-readable stderr banner (always — even if the file write fails).
   const why = sanitizeErr(stderr) || `exit ${code}`;
@@ -250,6 +300,8 @@ async function main() {
   const fullPrompt = COMPACT_PROMPT(task);
   const inChars = charCount(task);
   const callId = randomUUID().replace(/-/g, '').slice(0, 8);
+  // Predictable status path so the orchestrator can poll it (next to the call file when given).
+  const statusFile = statusFilePath(callId, opts.callFile);
 
   // ---- 4. Walk the chain -----------------------------------------------------
   let lastErr = ''; // short, sanitized reason from the last backend that actually failed
@@ -293,6 +345,9 @@ async function main() {
 
     state.start({ id: callId, backend: be, model, rule: D_rule, inChars });
     const startMs = Date.now();
+    // Heartbeat: emit "still running" to stderr + status file while the (possibly slow) CLI works,
+    // so the orchestrator can tell alive-but-slow from dead. Always stopped below (every branch).
+    const stopHeartbeat = startHeartbeat({ callId, statusFile, backend: be, model, tier: D_tier, rule: D_rule, startMs });
     let res;
     try {
       res = await invoke(invokeCfg, fullPrompt, { model, tier: D_tier, addDir: opts.addDir });
@@ -307,6 +362,7 @@ async function main() {
     // fallback); the single fallback tally is counted once on the success hop via fallbackCount.
     if (res.quota) {
       lastErr = `quota/credit limit on '${be}'`;
+      stopHeartbeat({ state: 'failed', kind: 'quota', code: res.code, elapsed_ms: durMs });
       logFailure({ backend: be, model, tier: D_tier, rule: D_rule, code: res.code, durMs,
         stderr: res.stderr || 'quota/credit limit reached', kind: 'quota', callId });
       state.end({ id: callId, backend: be, model, rule: D_rule, code: res.code, durMs, outChars, fallback: 0 });
@@ -317,9 +373,11 @@ async function main() {
     // cached health so a later call re-probes it instead of trusting a now-stale "healthy" verdict.
     if (res.code !== 0 || !cleanOut) {
       lastErr = sanitizeErr(res.stderr) || `exit ${res.code}, empty output`;
+      const kind = res.code !== 0 ? 'nonzero-exit' : 'empty-output';
+      stopHeartbeat({ state: 'failed', kind, code: res.code, elapsed_ms: durMs });
       logFailure({ backend: be, model, tier: D_tier, rule: D_rule, code: res.code, durMs,
         stderr: res.stderr || `no usable output (exit ${res.code})`,
-        kind: res.code !== 0 ? 'nonzero-exit' : 'empty-output', callId });
+        kind, callId });
       invalidateHealth(beCfg);
       state.end({ id: callId, backend: be, model, rule: D_rule, code: res.code, durMs, outChars, fallback: 0 });
       fallbackCount++; continue;
@@ -330,6 +388,7 @@ async function main() {
     // note. Missing/zero rate -> 0 cost.
     const rate = Number(beCfg.cost_per_1k_chars) || 0;
     const costMicros = Math.round(rate * (outChars / 1000) * 1e6);
+    stopHeartbeat({ state: 'done', code: 0, elapsed_ms: durMs, out_chars: outChars });
     state.end({ id: callId, backend: be, model, rule: D_rule, code: 0, durMs, outChars, fallback: fallbackCount, costMicros });
     process.stdout.write(cleanOut + '\n');
     process.exit(0);