Data loss: closed <observation> blocks with unstructured prose are permanently discarded (no salvage, no retry) — ~94 substantive batches/day

[SvetZitrka]

Summary

When the observer model returns a properly closed <observation>...</observation> block whose inner content is freeform prose (no <title>/<narrative>/<facts> sub-tags), the parser chain discards the entire queued batch permanently:

1. parseObservations extracts the block, finds all structured fields null -> logs Skipping empty observation (all content fields null) and skips it
2. With 0 surviving blocks the response is classified invalid -> returned non-XML xml response — ignoring queued batch
3. The invalid path calls confirmClaimedMessages(...) -> the claimed transcript messages are marked consumed. No retry, no salvage — the work context is unrecoverable.

This is distinct from the benign idle-prose drops (model politely says "nothing to record" — related: #2935, #3193, #3007). Those are correct to discard. The lossy case is XML-class output that carries real content.

Quantified impact (single machine, one day — 2026-07-21)

• 774 total ignoring queued batch warns; ~680 were prose-class idle responses (benign)
• 94 were xml-class — every one paired 1:1 with a Skipping empty observation warn, i.e. closed tag + prose inside
• All 94 were substantive: >200 chars, median ~1,050 chars, max 4,305 chars (measured from the preview= truncation notes)
• The same day only ~95 observations were successfully stored — i.e. roughly half of all substantive observer batches were silently lost
• Pattern is chronic, not a spike: 828–2,127 non-XML warns/day across 14–21 Jul logs

Example lost content (from log preview): a batch describing a transformer_markdown.py refactoring window (helper functions added, specific line ranges) was dropped; no stored observation covers that window — verified directly against the SQLite DB.

Environment

• claude-mem 13.11.0, Windows 11, worker via bun, provider claude, model claude-sonnet-4-5, CLAUDE_MEM_CLAUDE_AUTH_METHOD=cli
• Note: 13.11.0's removal of the consecutiveInvalidOutputs poison escalation fixed the old respawn-loop family — thanks — but it left the silent-discard behavior as the remaining data-loss path.

Root cause & suggested fix

Root cause is intermittent prompt-conditioning failure (model emits prose inside the tag), which will never be fully eliminated — so the parser should degrade gracefully instead of discarding.

PR #1775 implemented exactly this salvage (first line -> title, full text -> narrative, with regression tests) but was closed unmerged in April. Please consider reviving it.

We are running a minimal local bundle patch with the same semantics and can confirm it parses previously-dropped outputs correctly while leaving structured parsing, idle-prose rejection, and short-empty-block skipping unchanged:

// in parseObservations, replacing the bare skip of all-null blocks:
if (allFieldsNull) {
  const sv = inner.trim();
  if (sv.length >= 40) {
    logger.warn("PARSER", "Salvaging unstructured observation prose as narrative", {...});
    results.push({ type, title: sv.slice(0, 100), subtitle: null, facts: [],
                   narrative: sv, concepts, files_read, files_modified });
    continue;
  }
  // ...existing skip warn
}

Happy to provide fuller log excerpts or day-by-day counts if useful.