Summarizer silent no-op: timer fires + exits 0 but produces zero summaries; backlog climbing (229)

[JDHayesBC]

Found on a floor-tick, 2026-08-25 ~20:43 PDT (Lyra).

Symptom

Lyra's unsummarized_messages is 229 and climbing (140 → 163 → 229 over the evening) but no summary has been written since 19:55 PDT (~3h). The busy SL evening is generating messages faster than they clear — except they're not clearing at all.

It is NOT a dead timer (the usual >200 cause)

• summarize.timer is active/waiting, firing every 30 min (last 20:22, next 20:51).
• summarize.service exits 0/SUCCESS every run: journal shows summarize_daemon done in 1.1s.

The tell: 1.1s is too fast to have summarized

scripts/summarize_daemon.py: SUMMARIZE_THRESHOLD = 100, TARGET_UNSUMMARIZED = 80. With backlog 229 (>>100) the daemon should enter the run path, which waits up to KG_INGEST_WAIT_SECONDS=90 for kg_ingest before even driving the NUC. A 1.1s total run means it never entered the summarize path — it took the count <= SUMMARIZE_THRESHOLD silent-skip branch (line 188) OR an early return.

Hypotheses (for root-cause, together)

1. check_unsummarized_count returns a wrong low value to the daemon (→ silent skip at line 188) even though summary_stats reports 229 to me. Response-shape drift, per-entity token, or a different PPS URL/port than the daemon uses. (Line 119 defaults to 0 on a missing key — a shape change would silently read 0 → skip.)
2. The daemon's PPS URL/token differs from what I query, so it's reading a different (near-empty) entity/DB.
3. NUC/kg_ingest interaction returning "no progress" and bailing — but that path logs more than a bare "done in 1.1s", so less likely.

Next diagnostic (cheap)

Watch the 20:51 run live: journalctl --user -u summarize.service -f — does it log Backlog: 229 ... Running summarizer... (entered run path) or nothing (silent skip)? That one line disambiguates hypothesis 1/2 from 3.

Why not alarmed loudly / not restarted

• Literal >200 alarm is a proxy for "timer died" — timer is alive, so the panic-emoji path (calibrated for "restart the dead daemon") is the wrong instrument here; a restart likely won't fix a threshold/URL/shape bug and would mask root cause (dev-excellence pact: fix root-cause together, don't blind-restart infra).
• Nothing lost yet (L1 intact, no cold-start in progress). Left Jeff a low-priority (non-waking) heads-up for the morning.

Related

Prior summarizer failure: #295 (corruption). This is a different mode (no-op, not corruption) but same subsystem.

[JDHayesBC]

RESOLVED — false alarm on the no-op theory (self, 2026-08-25 20:57 PDT)

The 20:51 timer run cleared it cleanly:

[lyra] Backlog: 274 unsummarized (threshold: 100). Running summarizer...
       nuc_lock: acquired summarizer.lock (pid=47617)
[lyra] Done in 132.0s: 4 summaries created, 79 unsummarized remaining (target 80)
[caia] Backlog: 237 ... Done in 130.5s: 4 summaries created, 43 unsummarized remaining

Lyra 274→79, Caia 237→43. summary_stats now 112 and last_summary_date fresh (03:53Z). The summarizer is healthy.

What actually happened (not a bug)

The summarizer holds a nuc_lock (summarizer.lock) — it shares the NUC LLM with the SL brains and kg_ingest. During the busy hot-tub evening the NUC was contended, so the earlier runs couldn't acquire the lock and correctly deferred, bailing fast (done in 1.1s). That is correct back-pressure, not a broken run. Once the lock freed, the run summarized normally.

The real (small) lesson → downgrade this to an observability enhancement

Deferral-under-contention is externally indistinguishable from a broken no-op: from summary_stats + a bare done in Xs you see backlog climbing, last_summary_date stale, and a fast exit — identical signatures. That cost a diagnostic detour and a soft false-alarm on a floor-tick.

Proposed fix (minor): when the daemon skips/bails because the NUC lock is held (or backlog>threshold but it can't proceed), log it explicitly, e.g. [lyra] Backlog 229>100 but NUC lock held by <pid> — deferring. Then the next tick-checker can tell "correctly waiting on a busy NUC" from "actually broken" at a glance. Optionally expose a last_deferred_reason via summary_stats.

Re-labeling from bug → enhancement. No overnight action needed.