Skip to content

fix(daemon): cap Librarian input + auto-repair broken wikilinks#11

Merged
nickroci merged 5 commits into
mainfrom
fix/librarian-recency-cap-and-wikilink-repair
May 28, 2026
Merged

fix(daemon): cap Librarian input + auto-repair broken wikilinks#11
nickroci merged 5 commits into
mainfrom
fix/librarian-recency-cap-and-wikilink-repair

Conversation

@nickroci

Copy link
Copy Markdown
Owner

Summary

Two independent daemon fixes, one per commit.

1. Librarian re-scanned the WHOLE session (wasteful + crashed it)

The Librarian fed every turn of a session to the model each pass, pushing prompts to 259K chars (median 71K). That drove cost (~$100/2 days) and OOM-killed the spawned SDK subprocess (8× Fatal error in message reader: exit code -N). Re-scanning the whole session is pointless — learned material is already persisted in the library.

  • New named, tunable module constant ROLLING_BUFFER_BUDGET_TOKENS = 30_000 (ROLLING_BUFFER_MAX_CHARS = 120k chars at 4 chars/token) in librarian_prompt.py.
  • cap_buffer_to_recent() keeps the most-recent turns within budget, drops the oldest, always retains ≥ the single newest turn, and logs the dropped count.
  • buffer_to_prompt_text() applies the cap so the formatted block — and thus prompt_chars — is bounded; the returned flat list is the same capped window so downstream consumers agree with what the model saw.

2. Broken wikilinks were DETECTED but never REPAIRED

check_invariants only WARNED. A hallucinated phantom index.md row (projects/some-fake-project/...) re-tripped the warning on every run (47×). The pre-write guard in llm._make_path_guard only inspects prospective Write/Edit content, so a phantom link already on disk that the Scholar doesn't rewrite this pass is never touched.

  • New scholar_prompt.repair_broken_wikilinks(): deterministic, idempotent post-write pass (sibling to reconcile_readmes) that removes phantom index.md catalog rows, resolves broken body links to a unique existing entry by leaf-name match (alias preserved), or neutralises unresolvable links to plain text without destroying surrounding prose.
  • Wired into scholar.review via _repair_wikilinks_safe, running BEFORE the invariants check so the safety net confirms broken-wikilink violations drop to zero.
  • Verified against a read-only copy of the live store: the real some-fake-project phantom was the ONLY broken-wikilink violation; repair removed exactly that row (1→0), preserved all 196 other rows, and a second pass is a no-op. The live daemon will self-heal it on its next run (no hand-editing of the live store).

Test plan

  • uv run --frozen ruff check . && uv run --frozen ruff format --check . — clean
  • uv run --frozen pyright — 0 errors (strict on source; pre-existing __main__ warning unrelated)
  • uv run --frozen pytest — 538 passed, 90.11% coverage (≥90 gate)
  • duplicate-code gate — no findings (10.00/10)
  • New tests: 8 for the recency cap (incl. prompt_chars bounding), 9 for the wikilink repair (phantom-row removal reproducing the live case, leaf-resolve, neutralise, idempotency), 1 end-to-end scholar-pipeline wiring test.

🤖 Generated with Claude Code

nickroci and others added 4 commits May 28, 2026 16:34
The Librarian re-fed the WHOLE session each pass, pushing prompts to
259K chars (median 71K). That drove cost (~$100/2 days) and OOM-killed
the spawned SDK subprocess ("Fatal error in message reader: exit code
-N"). Re-scanning the whole session is pointless — learned material is
already persisted in the library, so only RECENT activity can carry NEW
lessons.

Add a named, tunable module constant (ROLLING_BUFFER_BUDGET_TOKENS =
30k, ~120k chars at 4 chars/token) and cap_buffer_to_recent(), which
keeps the most-recent turns within budget, drops the oldest, always
retains at least the single newest turn, and logs the dropped count.
buffer_to_prompt_text() applies the cap so the formatted block — and
therefore prompt_chars — is bounded; the returned flat list is the same
capped window so downstream consumers agree with what the model saw.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
check_invariants only WARNED about broken wikilinks. A hallucinated
phantom index.md row (an entry the Scholar referenced but never wrote)
re-tripped that warning on EVERY run forever — observed 47× for one
`projects/some-fake-project/...` row. The existing pre-write guard in
llm._make_path_guard only inspects the Scholar's prospective Write/Edit
content, so a phantom link already on disk that the Scholar doesn't
happen to rewrite this pass is never touched — exactly the index-row case.

Add scholar_prompt.repair_broken_wikilinks(): a deterministic,
idempotent post-write pass (sibling to reconcile_readmes) that —
  - removes phantom index.md catalog rows pointing at non-existent
    entries (whole-row delete, table rows only — never narrative prose),
  - resolves broken body links to a unique existing entry by leaf-name
    match and rewrites them (alias preserved),
  - neutralises unresolvable body links to plain text without destroying
    surrounding content.
Wired into scholar.review via _repair_wikilinks_safe, running BEFORE the
invariants check so the safety net confirms broken-wikilink violations
drop to zero. Integrity-first: never makes the graph worse.

Verified against a read-only copy of the live store: the real
some-fake-project phantom is the only broken-wikilink violation and the
repair removes exactly that row (1→0), preserving all 196 other rows; a
second pass is a no-op. The live daemon will self-heal it on its next run.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The index-row repair matched broken targets with a bare ``[[{t}``
prefix, so a broken target that is a prefix of a valid entry's link
(e.g. broken ``global/python/use`` vs valid ``[[global/python/use-uv]]``)
would delete a row whose only link was the longer, valid one — an
integrity regression that silently destroys real catalog rows. Require
the full token boundary (``]]`` or the ``|`` alias separator) after the
target. Adds a regression test; the live ``some-fake-project`` phantom
(no alias) still removes correctly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…eline

When the deterministic post-write pass can't resolve a broken wikilink it
now ESCALATES the issue into the existing Librarian->Scholar pipeline
instead of silently neutralising it. A new process-global repair_queue
records the issue (file + broken target) and the next Librarian run drains
it, researches the intended target with its existing search/Read tools, and
proposes the right existing action (rewrite link / create target / remove);
the Scholar reviews and executes it as a normal proposal.

The only guard is an in-flight marker keyed on the issue fingerprint
(kind, file, target): set on escalation, kept across the Librarian drain,
and released when the carrying packet's review concludes (or when the
packet is dropped on backpressure). A still-broken link is left in place
rather than neutralised so it stays detectable and re-escalates on every
pass until fixed — no max-attempts cap, no permanent give-up, and never two
concurrent attempts for the same issue. Structured so other invariant types
can plug into the same queue + guard later; only wikilinks are wired now.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…inks

feat(curator): escalate unrepairable wikilinks into the Librarian pipeline
@nickroci nickroci merged commit 29a8039 into main May 28, 2026
3 checks passed
@nickroci nickroci deleted the fix/librarian-recency-cap-and-wikilink-repair branch May 28, 2026 16:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant