Observation generation stuck in a self-sustaining poison→respawn loop: benign "prose"/"idle" SDK output counts toward the poison threshold, wiping context and dropping all captured work

[SvetZitrka]

Environment

• claude-mem 13.8.0 (observed; appears to be a regression vs 13.6.2, which captured tool work fine)
• Windows 11, bun runtime, worker HTTP on 127.0.0.1:37777
• Generator model: claude-sonnet-4-5

Summary

Across an entire multi-hour session, the observer/generator captured only user prompts and produced essentially zero observations of actual work (every generated observation had files_read: []). The worker logs show the generator session being "poisoned" and respawned every ~30–60s for hours.

Root cause: the parser treats a benign generator response (the model saying "nothing to record yet" as plain prose, or returning an empty/idle response) as an invalid output. After Y9 = 3 consecutive such outputs the session is "poisoned": conversationHistory = [] is wiped and the SDK session respawns (losing context, Issue #817, preservedPending: 0). Because the wipe destroys the very context the observer needs to accumulate, the next checkpoint again looks like "just a user prompt → nothing to record → prose", and the loop sustains itself forever. No work is ever captured.

Code (worker-service.cjs, function Bc)

// M9(t): output is "valid" only if it contains <observation>/<summary>/<skip_summary/>
// D9(t): outputClass -> "idle" | "poisoned" (matches Cwe API-failure markers) | "xml" | "prose"
if (!d.valid) {
  let k = D9(t), I = N9(t);
  if (e.consecutiveInvalidOutputs = (e.consecutiveInvalidOutputs ?? 0) + 1,
      g.warn("PARSER", `${a} returned non-XML ${k} response — ignoring queued batch`, {...}),
      k === "poisoned" || e.consecutiveInvalidOutputs >= Y9) {   // <-- BUG
        // poison: respawnPoisonedSession -> conversationHistory=[] wipe
  }
  ...
}

The problem is || e.consecutiveInvalidOutputs >= Y9: benign prose/idle (the model legitimately having nothing to record) accumulates toward the threshold and triggers a context-destroying poison. Cwe (the genuine poison markers) are real API failures ("context window", "session exhausted", "prompt is too long", …) — prose/idle are not failures.

Evidence (logs)

# Every cycle, for hours:
[WARN ] [PARSER] SDK returned non-XML prose response — ignoring queued batch {outputClass=prose, consecutiveInvalidOutputs=3}
[ERROR] [SESSION] SDK session poisoned — killing and respawning {outputClass=prose, consecutiveInvalidOutputs=3, threshold=3}
[WARN ] [SESSION] Respawning poisoned SDK session, preserving pending messages {preservedPending=0}
[WARN ] [SESSION] Discarding stale memory_session_id from previous worker instance (Issue #817)
# prose previews show the model only ever sees the prompt:
"I'm observing the primary session, but I don't see any technical work being performed yet - only the user's request."
# Net result: only prompt-echo observations, all with files_read: []

Suggested fix (one condition)

Only poison on a genuine API-failure marker, never on accumulated benign prose/idle:

-  k === "poisoned" || e.consecutiveInvalidOutputs >= Y9
+  k === "poisoned"

(Or, more conservatively: do not increment consecutiveInvalidOutputs for k === "prose" || k === "idle"; treat them as a no-op skip, equivalent to <skip_summary/>.)

Verification of the fix (applied locally, before/after on the same session 327)

	Before patch	After patch
Poison events	every ~30–60s for hours	none (count frozen)
consecutiveInvalidOutputs=3/4	→ poison + context wipe	ignored, session survives
Observations stored	1 in hours (prompt echo)	3 in minutes, real content
files_read	[]	populated (actual files read)

After the patch, the generator survives the early "idle/prose" checkpoints, accumulates the real tool activity, and emits + stores genuine <observation> XML ([DB] STORING | obsCount=1).

Related

Distinct from the worker-recycle/zombie issue (#3031); this is the observation-generation pipeline. Also touches Issue #817 (SDK context lost on respawn) — the poison loop makes #817 fire continuously.

[crippledgeek]

Confirming this still reproduces on 13.8.0, and adding a Linux data point (the existing reports here and in #2935 are Windows/macOS).

Env: claude-mem 13.8.0, Arch Linux (kernel 7.0.11), bun 1.3.14 / node v22.22.1, mode code.

Repro: a low-signal session (a couple of conversational/verification prompts, no heavy tool work) walks itself straight to the threshold and poisons — exactly the self-sustaining loop described above. Verbatim transcript for one session:

[PARSER]  SDK returned non-XML idle response — ignoring queued batch {outputClass=idle, consecutiveInvalidOutputs=1}
[SDK]     ← Response received (96 chars) (No observations - no tool execution logs or concrete technical work documented in the session.)
[PARSER]  SDK returned non-XML prose response — ignoring queued batch {outputClass=prose, consecutiveInvalidOutputs=2}
[PARSER]  SDK returned non-XML idle response — ignoring queued batch {outputClass=idle, consecutiveInvalidOutputs=3}
[ERROR][SESSION] SDK session poisoned — killing and respawning, pending messages preserved {consecutiveInvalidOutputs=3, threshold=3}
[WARN ][SESSION] Respawning poisoned SDK session, preserving pending messages {preservedPending=0}
[SESSION] Generator exited — finalizing session {reason=poisoned}

The "96 chars" prose is the observer correctly deciding there's nothing durable to record — i.e. a valid skip being scored as an invalid output, just as you diagnosed. Independently confirms your code reads (M9 validity, D9 class, Bc gate on || e.consecutiveInvalidOutputs >= Y9, Y9 = 3).

Worth underscoring the prompt side too (cf. #2935): the code mode's prompts.skip_guidance literally instructs "return an empty response only", while <skip_summary/> — the only skip token M9 accepts — appears in no code* mode JSON (only meme-tokens.json / email-investigation.json). So both your suggested fixes are needed-or-either: the one-line parser guard (k === "poisoned" only / don't count prose+idle) is the robust fix; aligning skip_guidance to emit <skip_summary reason="..."/> fixes the contract drift at the source.

+1 on the parser-side fix — it's platform-independent and stops the context-wipe regardless of what the model emits on an idle batch.

[hafizhsiraji]

Confirming this also reproduces on claude-mem 13.8.1, Windows 11 — adding another data point alongside the Linux one above.

Setup: project with low tool-work signal in a quiet session (mostly conversational/brainstorming, light tool use). Logs matched exactly:

[WARN] [PARSER] SDK returned non-XML prose response — ignoring queued batch {outputClass=prose, consecutiveInvalidOutputs=3}
[ERROR] [SESSION] SDK session poisoned — killing and respawning, pending messages preserved {outputClass=prose, consecutiveInvalidOutputs=3, threshold=3}

What I tried before finding this issue:

• Restarting the worker process — no effect, loop resumed within ~6s of the new worker spawning (poison count went 7→36→44 in ~3 min, then 54→65 after restart).
• Confirmed the issue isn't in-memory worker state — it's the same low-signal session getting re-poisoned immediately on respawn, matching the self-sustaining loop described above.

Workaround I'm using in the meantime: CLAUDE_MEM_EXCLUDED_PROJECTS to skip observation capture for the affected project entirely (loses memory for that project, but stops the resource burn).

+1 on the parser-side fix in #3059 — looking forward to it landing so I can drop the workaround.