Skip to content

feat(daemon): typed-output curator over the subscription SDK (Pydantic-AI ergonomics, no metered API)#16

Merged
nickroci merged 8 commits into
mainfrom
feat/typed-agent-shim
May 29, 2026
Merged

feat(daemon): typed-output curator over the subscription SDK (Pydantic-AI ergonomics, no metered API)#16
nickroci merged 8 commits into
mainfrom
feat/typed-agent-shim

Conversation

@nickroci

@nickroci nickroci commented May 29, 2026

Copy link
Copy Markdown
Owner

Why

The daemon's Librarian + Scholar must authenticate via the Claude Code subscription (claude-agent-sdk), never the metered Anthropic API (see the reverted PR #14 — Pydantic-AI's provider silently switched billing to pay-per-token). But we still wanted Pydantic-AI's boundary discipline: typed, validated curator output with model self-correction, instead of scraping JSON out of free text and repairing it after the fact.

What

A small in-house shim, typed_agent.run_typed, gives that discipline on top of the subscription SDK:

  • The model "returns" by calling a submit_result MCP tool whose input_schema is the Pydantic output model (ScholarDecisions / LibrarianProposal) — the same way Pydantic-AI hands Anthropic a schema.
  • A validator runs at that boundary; malformed output (unresolvable wikilink, over-cap directory, unparseable frontmatter) is handed straight back as the tool result, so the model self-corrects in-band. No valid result within the retry budget → TypedAgentError → the batch is dropped (lessons recur). Bad data never crosses the boundary.
  • Forcing the tool call isn't possible (Anthropic can't force a specific tool alongside free research-tool use, and it's incompatible with extended thinking — Pydantic-AI falls back the same way), so "never submitted" is just a terminal validation failure.

Both roles are rewired onto it; the model writes nothing (read-only research tools only — _agent_research) and a deterministic executor is the sole writer. Reuses the ported pure modules (_schemas, _validation, scholar_executor); deletes _response_parser + the json-repair path; guts llm.py to constants + LLMTimeout. README architecture section updated (and two now-stale bullets — the can_use_tool path guard and the JSON extractor — corrected).

Tests & gates

  • 582 passed, 90.49% coverage (gate is 90%). ruff + pyright strict (0 errors) + pylint duplicate-code all clean. Added pytest-asyncio (auto mode) for the shim's async tests.
  • typed_agent 97%, with validator + research-tool + run-wrapper coverage for both roles; agent-failure paths (TypedAgentError / LLMTimeout / executor error) verified to drop the batch, never crash review.

How this was built (for review)

The shim keystone (typed_agent.py + tests) was hand-written and verified first. The mechanical port + rewire was done by a background agent, which hit a transient 529 mid-test-migration; I finished the test migration and restored coverage, and reviewed the rewiring (validators, run-wrappers, executor seam) while writing its tests. Live smoke test confirmed working.

🤖 Generated with Claude Code

nickroci and others added 8 commits May 29, 2026 01:11
@nickroci @claude
feat(daemon): typed-agent shim over claude-agent-sdk (typed output + …
c2ae4ca 
…retry)

Keystone for keeping the daemon on Claude Code subscription auth while
recovering the one Pydantic-AI behaviour we want: typed, validated output with
model self-correction — no hand-scraped JSON, no post-hoc repair.

`run_typed(prompt, OutputType, deps=..., validators=...)` registers a
`submit_result` in-process MCP tool whose `input_schema` is
`OutputType.model_json_schema()` (the same shape Pydantic-AI hands Anthropic).
The handler runs `model_validate` + the supplied validators; on success it
stashes the typed object, on failure it returns the error in-band (`is_error`)
so the agentic loop continues and the model re-emits. No valid result within
the retry budget -> TypedAgentError, and the caller drops the batch. Bad data
never crosses the boundary.

Forcing the submit call isn't possible (Anthropic can't force a specific tool
alongside free research-tool use, and forcing is incompatible with extended
thinking — Pydantic-AI falls back the same way), so "never submitted" is just a
terminal validation failure.

Not yet wired into the Scholar/Librarian — that follows once the schemas and
validators are ported. typed_agent.py at 97% line coverage (the remaining line
is the live SDK handler dispatch, covered by the upcoming smoke test).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@nickroci @claude
feat(daemon): rewire Librarian + Scholar onto the typed-output shim
9f03807 
Replace the old Claude-Agent-SDK free-text + json-repair path with the
typed_agent.run_typed shim (still claude_agent_sdk under the hood — the
subscription backend, never the metered API). Each role returns a typed,
boundary-validated result and writes nothing to disk.

- _agent_research.py (new): in-process SDK MCP server exposing ONLY the
  four read-only research tools (read_entry, grep_library, bm25_search,
  embedding_search), scoped to the knowledge root. Mirrors the read-only
  surface the reverted migration's _agent_common registered; no
  move/write tool is ever wired. research_server_and_tools() returns the
  (mcp_servers, allowed_tools) pair a role hands to run_typed.
- library_tools.py: add public run_bm25_search / run_embedding_search /
  move_entries / unwrap_text_response / text_response wrappers the research
  server and the Scholar executor call in-process. Keep the SDK search
  server intact.
- scholar.py: build the prompt (scholar_prompt.build_prompt), call
  run_typed(ScholarDecisions, model=claude-opus-4-7, …) with the read-only
  tools + the ported validate_decisions whole-batch validator (ctx.deps →
  deps, pydantic_ai.ModelRetry → typed_agent.ModelRetry), then apply
  result.output via scholar_executor. PRESERVES the reconcile / wikilink-
  repair / invariant / escalation safety net and stats. On TypedAgentError
  (no valid result in budget) the batch is dropped exactly like a failed
  run.
- librarian.py: call run_typed(LibrarianProposal, model=claude-sonnet-4-6,
  …) with the read-only tools + the ported validate_proposal boundary
  validator, and emit the typed output into the proposals/interrupts
  packet. PRESERVES the rolling-buffer recency cap, project-bucket
  derivation, and integrity-repair drain + fingerprint escalation.
- llm.py: drop run_scholar_call / run_librarian_call, the free-text JSON
  drain, and the can_use_tool path guard (the model writes nothing now);
  keep LLMTimeout + the model/timeout constants.
- _response_parser.py: deleted; json-repair dependency removed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@nickroci @claude
wip(daemon): finish curator test migration; suite green (542 pass)
e5ee8f9 
Completes the test-migration tail the background agent was cut off on (it hit a
transient 529). Deletes the now-obsolete test_llm.py — llm.py is reduced to
LLMTimeout + model/timeout constants, so its tests for run_scholar_call /
_drain_query / _check_and_repair_writes / the path guard reference removed code
(matches the already-deleted test_llm_path_guard / test_response_parser).

Full daemon suite passes (542). Coverage is 85% < the 90% gate: the rewired
scholar.py is at 63% — tests for the run_typed call path + TypedAgentError drop
handling land next. No pydantic_ai, no ANTHROPIC_API_KEY (subscription only).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@nickroci @claude
test(daemon): add pytest-asyncio; run shim tests natively
df52523 
Add pytest-asyncio (dev) + asyncio_mode="auto" so async tests run without the
asyncio.run() wrappers I'd used to dodge the missing dep. Refactor
test_typed_agent.py to native `async def` tests. No other async tests exist, so
auto mode is a no-op elsewhere; full suite still 542 passed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@nickroci @claude
test(daemon): cover the typed-curator rewiring (back to 90%)
41b219a 
The background agent rewired Scholar + Librarian onto the shim but didn't port
their tests, dropping coverage to 85% (scholar 63%, librarian 47%,
_agent_research 19%). Restore the boundary-validator + run-wrapper + research-
tool coverage the gate needs:

- test_agent_research.py: the read-only tools (inside/read_entry/grep/search) +
  _coerce_k + server/tool-ref wiring.
- test_scholar_validators.py: validate_decisions + wikilink/flat-dir-cap helpers
  across all action kinds; run_scholar_agent (stubbed run_typed) + timeout.
- test_librarian_validators.py: validate_proposal (path well-formedness + body
  frontmatter) across all action kinds; run_librarian_agent + timeout.
- test_scholar.py: agent-failure drop paths (TypedAgentError / LLMTimeout /
  executor error) never crash review.

Validator tests adapted from the reverted migration's test_scholar_agent.py
(now scholar/librarian modules, typed_agent.ModelRetry). Full suite: 582 passed,
90.49% coverage.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@nickroci @claude
docs(readme): document the typed-output shim + fix stale curator bullets
1e8bdaf 
Adds a "Typed output over the subscription SDK" note to the architecture
section describing typed_agent.run_typed (submit_result tool whose schema is
the Pydantic model + in-band validator retry; subscription auth, no metered
API). Also corrects two bullets the migration invalidated: the can_use_tool
path guard and the final-{...}-block JSON extractor are gone (the model writes
nothing now; a deterministic executor is the sole writer).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@nickroci @claude
docs(readme): clarify subscription-based cost; refresh status counts
22bc178 
The curator runs on the Claude Code subscription (claude-agent-sdk), not a
metered API — so reframe "A note on cost" around Max/Pro quota (token-heavy,
counts against your plan; rate-limit drops a batch rather than billing a paid
API), and note the same in the intro. Per-token $ figures kept only for the
metered-key case. Refresh Status test counts (516->582 daemon, 824 total) and
note the curator now runs through the typed-output shim.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@nickroci @claude
fix(daemon): restore CLAUDE_INVOKED_BY recursion guard on curator SDK…
b3a9fc8 
… calls

CRITICAL: the typed-agent rewiring dropped the recursion-guard env the old
llm.py set on every SDK call. The curator's claude-agent-sdk calls were running
with no CLAUDE_INVOKED_BY, so the Claude processes the SDK spawns ran the user's
hooks normally -> appended to events.jsonl -> the daemon ingested its own model
calls -> curator ran again -> infinite loop (observed: librarian queue saturated,
24k+ self-generated events, runaway CPU + subscription-quota burn; the flood
events carried cwd=~/.agent-mem/knowledge with no guard marker).

Restore recursion_guard_env() (inherited env + CLAUDE_INVOKED_BY=agent_mem_daemon)
in llm.py and pass it from both run_scholar_agent / run_librarian_agent into
run_typed. Adds a regression test per role asserting the guard env is set, so
this can't silently regress again.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@nickroci nickroci merged commit e135cd1 into main May 29, 2026
4 checks passed
@nickroci nickroci deleted the feat/typed-agent-shim branch May 29, 2026 13:21
This was referenced May 29, 2026
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@nickroci