fix(observer): window invalid-output respawn so benign idle can't poison quiet sessions

[crippledgeek]

What

Replaces the unbounded consecutiveInvalidOutputs >= 3 respawn counter — which wrongly counted benign idle/prose SDK output and poison-looped quiet sessions, wiping context and dropping all captured work — with a time-windowed burst counter (modelled on systemd's StartLimitBurst/StartLimitIntervalSec and Erlang/OTP supervisor intensity/period).

Closes #3032
Refs #2935 #3007 #3037 #3022 #2955 #2960

Why

On low-signal sessions the SDK legitimately emits idle/empty/prose responses. The old counter treated every one as an "invalid output" and, after 3 in a row, killed and respawned the observer — which on a quiet session just produces more idle output, so the respawn itself re-triggers the counter: a self-sustaining poison→respawn loop. The fix only respawns when invalid outputs arrive as a burst within a bounded window, so isolated benign idles can never accumulate to the threshold.

How

• New module src/services/worker/agents/respawn-policy.ts: evaluateRespawn (windowed decision with an exhaustiveness guard over output classes), parseRespawnPolicy, cached settings-backed getRespawnPolicy, and a FailureWindow class.
• ActiveSession.consecutiveInvalidOutputs → invalidOutputWindow (timestamps within the window, not a monotonic count).
• poisoned output still respawns immediately; idle is exempt by default.
• Respawn-policy telemetry dimension wired through the scrubber; an orphaned telemetry key retired.

New settings (all string, in SettingsDefaultsManager, defaults reproduce prior behavior minus the bug)

Key	Default	Range
CLAUDE_MEM_INVALID_OUTPUT_EXEMPT_CLASSES	idle	comma-list of output classes
CLAUDE_MEM_INVALID_OUTPUT_RESPAWN_THRESHOLD	3	1..=100
CLAUDE_MEM_INVALID_OUTPUT_WINDOW_MS	60000	1000..=3_600_000

Tests

• tests/worker/agents/respawn-policy.test.ts (new, comprehensive: windowing, threshold boundaries, exempt classes, exhaustiveness).
• tests/shared/settings-respawn-policy.test.ts (new, settings parse + validation bounds).
• Extended tests/worker/poison-respawn.test.ts, response-processor.test.ts, scrub.test.ts.
• Targeted suite: 71 pass / 0 fail; tsc --noEmit clean.
• Live soak (prior session): real Haiku run emitted 5× idle + 3× prose → badCount ≤ 1, 0 poisons (old code would have respawned).

Notes

• Poison-loop cluster: this PR fully resolves Observation generation stuck in a self-sustaining poison→respawn loop: benign "prose"/"idle" SDK output counts toward the poison threshold, wiping context and dropping all captured work #3032; the other Refs (Generator poison loop: prompt says 'return empty response' to skip, but parser only accepts <skip_summary/> — empty/prose replies poison the session #2935, SDK session repeatedly poisoned by non-XML idle responses during heavy agentic sessions #3007, Generation halts in an infinite poison/respawn loop when the subscription account hits its weekly/usage limit (rate-limit prose misclassified as invalid output) #3037, Observer respawns ~1/3 of Haiku sessions on a keyword false-positive ("poisoned") — it should just ignore unparseable output #3022, SDK session poison-loop: generator emits prose "empty observation" instead of <skip_summary/>, triggering repeated kill/respawn #2955, tracking: observer poison loop — skip-prose counted as invalid, triggering endless kill/respawn cycle #2960) describe the same self-sustaining respawn loop from different triggers. Windowing the invalid-output burst is a contributing mitigation for them, so they're linked as related rather than closed — e.g. Observer respawns ~1/3 of Haiku sessions on a keyword false-positive ("poisoned") — it should just ignore unparseable output #3022's poisoned-keyword false-positive is a distinct concern the poisoned class still respawns on immediately.
• Independent of feat(worker): add CLAUDE_MEM_WORKER_SCRIPT_PATH to pin the worker bundle #3055/fix(chroma): pin chroma-mcp via uvx canonical tool@version form (#2939) #3057/test(settings): isolate loadFromFile tests from all ambient default env keys #3058 (branched cleanly off main, no shared commits) — can merge in any order.
• Pre-existing/flaky suite failures (SettingsDefaultsManager env-leak — addressed in test(settings): isolate loadFromFile tests from all ambient default env keys #3058 — and GeminiProvider/Worker-API full-suite-concurrency flakes) are unrelated; this diff touches none of those modules.

[greptile-apps]

Greptile Summary

This PR changes observer invalid-output recovery to use a time-windowed respawn policy. The main changes are:

• Adds a settings-backed respawn policy with exempt classes, burst threshold, and window duration.
• Replaces the session's unbounded invalid-output counter with a rolling failure window.
• Keeps poisoned output as an immediate respawn trigger while exempting idle output by default.
• Updates telemetry fields, documentation, and CLI output for the new respawn threshold key.
• Adds focused tests for policy parsing, window behavior, response processing, scrubbed telemetry, and poisoned-session respawn.

Confidence Score: 5/5

The change is narrowly scoped to observer recovery policy and telemetry/settings wiring, with focused tests covering the new windowed behavior and parsing bounds.

No correctness issues were identified in the reviewed changes, and the added tests exercise the main behavioral boundaries described by the PR.

T-Rex Logs

What T-Rex did

• Compared baseline respawn policy with the after-state to verify how idle invalidOutputWindow and consecutiveInvalidOutputs behave, noting that prose respawns on the third burst output and valid XML resets badCount to 0, while isolated prose events beyond the 60000ms window remain badCount 1 and poisoned output respawns immediately while preserving pending count and clearing the window.
• Validated the respawn-settings changes by comparing before and after artifacts, documenting defaults and overrides such as idle/3/60000, idle+prose/2/5000, invalid fallback idle/3/60000, and the env threshold override to 7.
• Checked how scrub output and ResponseProcessor changes affect what fields are preserved, confirming that respawn_threshold is present and consecutive_invalid_outputs is no longer part of the head output payload.

_{Ran code and verified through T-Rex}

_{Reviews (1): Last reviewed commit: "fix(observer): wire respawn telemetry di..." | Re-trigger Greptile}