feat(scholar): migrate curator to Pydantic AI with typed structured output#14
Merged
Conversation
Adds pydantic-ai-slim[anthropic]>=1.99 so the Scholar can move to a typed Pydantic AI agent. 1.99+ is required for compatibility with the installed anthropic>=0.97 (older pydantic-ai imports a UserLocation symbol that was renamed to BetaUserLocationParam). claude-agent-sdk is retained for the Librarian and the hook/flush paths. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the Scholar's claude-agent-sdk + hand-scraped-JSON path with a
Pydantic AI agent that returns typed, boundary-validated actions which a
deterministic daemon executor applies to disk. The fragile
extract_json_blob + json_repair scrape and model-written files are gone
from the Scholar path (the Librarian still uses _response_parser).
- scholar_agent.py: Agent on anthropic:claude-opus-4-7, output_type =
ScholarDecisions, retries={'output': 4}, defer_model_check. Read-only
in-process tools (read_entry / grep_library / bm25_search /
embedding_search) wrapping existing code — the model writes nothing.
An @output_validator does the context-dependent checks (every body
wikilink resolves to an existing entry or a batch-created path; no dir
left over MAX_FLAT_DIR_ENTRIES) and raises ModelRetry with a specific,
actionable message so the model fixes and re-emits.
- scholar_executor.py: deterministic apply of a validated ScholarDecisions
— write/update/merge/move/archive/deprecate, index.md upsert, log.md
append; moves reuse library_tools; failure-tolerant per action.
- scholar.review: run agent -> validated ScholarDecisions -> executor ->
existing reconcile/repair/invariant/escalation safety net (unchanged).
- scholar_prompt: instruct the model to verify via the read tools and
RETURN the six core typed actions; dropped the emit-JSON / write-files
instructions and the Scholar's use of _response_parser/ScholarReview.
- _resolve_wikilink (llm), check_invariants (scholar_prompt) now delegate
to the shared _validation helpers so boundary + post-write checks agree.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…layer Stage 2 of the curator migration: the Librarian is now a typed Pydantic AI agent mirroring the Scholar. It runs on anthropic:claude-sonnet-4-6, returns a validated LibrarianProposal (output_type), and uses the same read-only research tools (read/grep/bm25/embedding) — now shared with the Scholar via a new _agent_common module so there's no duplicate-code. A Librarian-specific output_validator bounces malformed proposals (absolute/escaping/non-.md paths, unparseable body frontmatter) back via ModelRetry; the bar is deliberately lower than the Scholar's (recall over precision — the Scholar stays the precision gatekeeper). librarian.scan dumps the typed result into the proposals/interrupts dict packet — no JSON scraping. The rolling-buffer recency cap, project-bucket derivation, and integrity-repair-task drain + fingerprint escalation are all preserved. With neither curator on the old path anymore, this removes the now-dead code: delete _response_parser (extract_json_blob/parse_response) and the json-repair dependency; gut llm.py down to LLMTimeout (drop run_librarian_call, run_scholar_call, _drain_query, _make_path_guard / _can_use_tool / _check_and_repair_writes and the allowed_tools machinery); drop the Claude-Agent-SDK MCP wrapper from library_tools (make_library_mcp_server + fully_qualified_* helpers) so the agents call the in-process search runners directly; and remove claude-agent-sdk from the daemon's deps (it survives only as a transitive dep of agent-mem-search). claude_agent_sdk, json_repair, and _response_parser are now fully gone from the daemon package. src/ is untouched. Gates from daemon/: ruff check + format clean; pyright strict 0 errors; pytest 592 passed at 90.89% coverage; duplicate-code clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…safe `load_pickled` only caught (OSError, UnpicklingError, EOFError, AttributeError, ModuleNotFoundError), so a corrupt pickle that surfaced as UnicodeDecodeError (a ValueError, not UnpicklingError) propagated instead of degrading to a rebuild. In the daemon this silently disabled embedding retrieval until the index was deleted by hand — priming fell back to BM25-only. Broaden to `except Exception` to honour the documented "any read failure -> rebuild" contract (BaseException still propagates). Root cause of the corruption: save_index wrote a shared `<name>.tmp` before the atomic rename, but the daemon saves from parallel Librarian threads (+ priming + prewarm). Concurrent rebuilds of a stale index interleaved into that one temp file, producing a half-written pickle. Add shared `bm25.save_pickled` that writes a unique temp file per writer (tempfile.mkstemp) then os.replace; route both BM25 and embedding save_index through it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
nickroci
added a commit
that referenced
this pull request
May 28, 2026
Revert PR #14 (Pydantic AI migration) — restore subscription auth, keep index hardening
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Migrates the Scholar curator from the Claude Agent SDK + hand-scraped JSON to Pydantic AI with typed structured output and boundary validation (Stage 1; the Librarian stays on the SDK for a later stage).
Agentonanthropic:claude-opus-4-7withoutput_type = ScholarDecisions(a union of the existing action models +interrupts_processed). It has read-only verification tools (read_entry,grep_library,bm25_search,embedding_search) wired as in-process@agent.tools — the model writes no files.idmatches the filename slug,scopeagrees with the path) plus an@agent.output_validatorfor context-dependent checks (every[[wikilink]]resolves to an existing entry or a path another action in the batch creates; no directory left overMAX_FLAT_DIR_ENTRIES). Failuresraise ModelRetry("<specific message>"); retry budgetretries={'output': 4}. Shared_validationhelpers keep the boundary checks and the post-writecheck_invariantssafety net in agreement.scholar_executor.py) applies a validatedScholarDecisionsto disk — write/update/merge/move/archive/deprecate,index.mdupsert,log.mdappend (moves reuselibrary_tools), failure-tolerant per action.scholar.reviewnow runs the agent → validatedScholarDecisions→ executor → the unchanged reconcile/wikilink-repair/invariant/escalation safety net.extract_json_blob/json_repair/_response_parser/ScholarReview._response_parserremains (the Librarian still uses it); a test asserts none of those tokens remain in the Scholar path.Deferred
llm.run_scholar_calland the Scholar-only pre-write wikilink guard (_check_and_repair_writesetc.) are now unused by the Scholar but left in place (still imported only withinllm.py; harmless, fully tested)._resolve_wikilinkwas made to delegate to the shared_validationhelper so the duplicate-code gate passes. Removing the rest of that dead chain (and its tests) is a clean, separable follow-up.Test plan
uv run --frozen ruff check . && uv run --frozen ruff format --check .uv run --frozen pyright(strict source; 0 errors)uv run --frozen pytest(638 passed, 90.5% coverage)FunctionModel); retry-budget exhaustion; executor applies each action type to a fixture library; end-to-endreviewapplies a validatedScholarDecisions; asserts nojson_repair/extract_json_blobin the Scholar path.🤖 Generated with Claude Code