fix(daemon): cap Librarian input + auto-repair broken wikilinks#11
Merged
Conversation
The Librarian re-fed the WHOLE session each pass, pushing prompts to
259K chars (median 71K). That drove cost (~$100/2 days) and OOM-killed
the spawned SDK subprocess ("Fatal error in message reader: exit code
-N"). Re-scanning the whole session is pointless — learned material is
already persisted in the library, so only RECENT activity can carry NEW
lessons.
Add a named, tunable module constant (ROLLING_BUFFER_BUDGET_TOKENS =
30k, ~120k chars at 4 chars/token) and cap_buffer_to_recent(), which
keeps the most-recent turns within budget, drops the oldest, always
retains at least the single newest turn, and logs the dropped count.
buffer_to_prompt_text() applies the cap so the formatted block — and
therefore prompt_chars — is bounded; the returned flat list is the same
capped window so downstream consumers agree with what the model saw.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
check_invariants only WARNED about broken wikilinks. A hallucinated
phantom index.md row (an entry the Scholar referenced but never wrote)
re-tripped that warning on EVERY run forever — observed 47× for one
`projects/some-fake-project/...` row. The existing pre-write guard in
llm._make_path_guard only inspects the Scholar's prospective Write/Edit
content, so a phantom link already on disk that the Scholar doesn't
happen to rewrite this pass is never touched — exactly the index-row case.
Add scholar_prompt.repair_broken_wikilinks(): a deterministic,
idempotent post-write pass (sibling to reconcile_readmes) that —
- removes phantom index.md catalog rows pointing at non-existent
entries (whole-row delete, table rows only — never narrative prose),
- resolves broken body links to a unique existing entry by leaf-name
match and rewrites them (alias preserved),
- neutralises unresolvable body links to plain text without destroying
surrounding content.
Wired into scholar.review via _repair_wikilinks_safe, running BEFORE the
invariants check so the safety net confirms broken-wikilink violations
drop to zero. Integrity-first: never makes the graph worse.
Verified against a read-only copy of the live store: the real
some-fake-project phantom is the only broken-wikilink violation and the
repair removes exactly that row (1→0), preserving all 196 other rows; a
second pass is a no-op. The live daemon will self-heal it on its next run.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The index-row repair matched broken targets with a bare ``[[{t}``
prefix, so a broken target that is a prefix of a valid entry's link
(e.g. broken ``global/python/use`` vs valid ``[[global/python/use-uv]]``)
would delete a row whose only link was the longer, valid one — an
integrity regression that silently destroys real catalog rows. Require
the full token boundary (``]]`` or the ``|`` alias separator) after the
target. Adds a regression test; the live ``some-fake-project`` phantom
(no alias) still removes correctly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…eline When the deterministic post-write pass can't resolve a broken wikilink it now ESCALATES the issue into the existing Librarian->Scholar pipeline instead of silently neutralising it. A new process-global repair_queue records the issue (file + broken target) and the next Librarian run drains it, researches the intended target with its existing search/Read tools, and proposes the right existing action (rewrite link / create target / remove); the Scholar reviews and executes it as a normal proposal. The only guard is an in-flight marker keyed on the issue fingerprint (kind, file, target): set on escalation, kept across the Librarian drain, and released when the carrying packet's review concludes (or when the packet is dropped on backpressure). A still-broken link is left in place rather than neutralised so it stays detectable and re-escalates on every pass until fixed — no max-attempts cap, no permanent give-up, and never two concurrent attempts for the same issue. Structured so other invariant types can plug into the same queue + guard later; only wikilinks are wired now. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
5 tasks
…inks feat(curator): escalate unrepairable wikilinks into the Librarian pipeline
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two independent daemon fixes, one per commit.
1. Librarian re-scanned the WHOLE session (wasteful + crashed it)
The Librarian fed every turn of a session to the model each pass, pushing prompts to 259K chars (median 71K). That drove cost (~$100/2 days) and OOM-killed the spawned SDK subprocess (8×
Fatal error in message reader: exit code -N). Re-scanning the whole session is pointless — learned material is already persisted in the library.ROLLING_BUFFER_BUDGET_TOKENS = 30_000(ROLLING_BUFFER_MAX_CHARS= 120k chars at 4 chars/token) inlibrarian_prompt.py.cap_buffer_to_recent()keeps the most-recent turns within budget, drops the oldest, always retains ≥ the single newest turn, and logs the dropped count.buffer_to_prompt_text()applies the cap so the formatted block — and thusprompt_chars— is bounded; the returned flat list is the same capped window so downstream consumers agree with what the model saw.2. Broken wikilinks were DETECTED but never REPAIRED
check_invariantsonly WARNED. A hallucinated phantomindex.mdrow (projects/some-fake-project/...) re-tripped the warning on every run (47×). The pre-write guard inllm._make_path_guardonly inspects prospective Write/Edit content, so a phantom link already on disk that the Scholar doesn't rewrite this pass is never touched.scholar_prompt.repair_broken_wikilinks(): deterministic, idempotent post-write pass (sibling toreconcile_readmes) that removes phantomindex.mdcatalog rows, resolves broken body links to a unique existing entry by leaf-name match (alias preserved), or neutralises unresolvable links to plain text without destroying surrounding prose.scholar.reviewvia_repair_wikilinks_safe, running BEFORE the invariants check so the safety net confirms broken-wikilink violations drop to zero.some-fake-projectphantom was the ONLY broken-wikilink violation; repair removed exactly that row (1→0), preserved all 196 other rows, and a second pass is a no-op. The live daemon will self-heal it on its next run (no hand-editing of the live store).Test plan
uv run --frozen ruff check . && uv run --frozen ruff format --check .— cleanuv run --frozen pyright— 0 errors (strict on source; pre-existing__main__warning unrelated)uv run --frozen pytest— 538 passed, 90.11% coverage (≥90 gate)prompt_charsbounding), 9 for the wikilink repair (phantom-row removal reproducing the live case, leaf-resolve, neutralise, idempotency), 1 end-to-end scholar-pipeline wiring test.🤖 Generated with Claude Code