PLAN.md

PLAN: Replace pi-ai / pi-agent-core with an in-repo LLM client + agent loop

Goal

Remove @mariozechner/pi-ai and @mariozechner/pi-agent-core from Mame and replace them with a small, owned OpenAI-compatible chat-completions client (src/llm/) and an owned agent loop. No client-side model catalog: Mame sends whatever model id the persona names and surfaces the provider's error verbatim if it's rejected.

Acceptance bar: a persona with model: openrouter/x-ai/grok-4.5 works end to end (reply + at least one tool round-trip), and nothing that worked on v0.3.x broke. Final version: 0.4.0.

Decisions (made, not open)

• Direct anthropic/ and google/ backends are dropped. Only two routes:
  • openrouter/<rest> → https://openrouter.ai/api/v1, OPENROUTER_API_KEY, model id <rest>.
  • Anything else → generic OpenAI-compatible escape hatch: MAME_LLM_BASE_URL + MAME_LLM_API_KEY; the full model string is sent as-is as the model id. If MAME_LLM_BASE_URL is unset for a non-openrouter model, fail with a clear error: "Direct anthropic/google backends were removed in 0.4.0 — use openrouter/anthropic/... or openrouter/google/..., or set MAME_LLM_BASE_URL / MAME_LLM_API_KEY for any OpenAI-compatible endpoint."
  • README and example persona snippets that reference google/gemini-* and anthropic/claude-* model ids get rewritten to openrouter/... equivalents (README.md lines ~233, ~286–287).
• @anthropic-ai/sdk is also removed. It is only used for types (ChatMessage/ContentBlock shapes) in src/model-router.ts, src/improve.ts, src/tools/index.ts — all of which are rewritten here. Net dependency count drops by 3.
• pi-ai's TypeBox re-exports (Type, Static, TSchema) are replaced with a direct dependency on @sinclair/typebox (it was already in the tree as a transitive dep; making it direct keeps the existing schema code in heartbeat.ts/onboard.ts almost unchanged). TypeBox schemas ARE JSON Schema, so they pass straight into the tools payload.

Out of scope (non-goals)

• No changes to vault, memory internals, gateway channels, or the MCP server / ask-human plumbing beyond import adjustments (src/mcp-server.ts, src/ask-human-state.ts untouched).
• No streaming support. No token/cost accounting. No prompt caching.
• No collapse-refactors beyond what the dependency removal forces.
• Do NOT restart the user's running systemd mame service. Live testing is via CLI one-shot only.

Current state (what exists, verified)

• src/agent.ts — think() builds a fresh pi-agent-core Agent per turn; conversation buffer of pi-ai Message[] keyed persona:channel:project (max 20); heartbeat channel bypasses the buffer; thinkingLevel resolution (turn override > model.reasoning→"medium" > "off"); image URLs fetched to base64; errors surfaced as friendly strings, daemon never crashes.
• src/model-router.ts — parseModelString() (google/ | openrouter/ | bare → anthropic) plus the chatCompletion() shim over pi-ai completeSimple with Anthropic-flavored ChatMessage/ModelResponse translation. Only remaining chatCompletion caller is src/improve.ts (no tools, text-only).
• src/tools/index.ts — two surfaces: legacy executeToolCalls/withRetry (now caller-less except via chatCompletion path — effectively dead) and getAgentTools() (pi AgentTool). Shared: isTransientError() (timeout/ econnreset/econnrefused by message; 429/502/503/504 by .status) and enforceApproval() — the v0.2.0 approval gate: runs once per flagged call, before/outside the p-retry budget, fails closed on deny/timeout/undeliverable.
• src/heartbeat.ts — parseScheduleWithModel() forces a submit_schedule tool call (TypeBox ScheduleSchema) via a single-tool Agent; empty-string project coerced to null; returns [] on any parse failure. No unit tests today.
• src/onboard.ts — runOnboardingConversation() drives an Agent turn-by-turn with inline AgentTools (write_config, set_secret, + signal-only tools) closing over send().
• src/magazine/digest.ts — also imports pi-ai (completeSimple, getModel) for text-only llmJson/llmText helpers. Must migrate (not named in the original request, but the dep can't be removed otherwise).
• src/improve.ts — text-only chatCompletion call, best-effort.
• Tests: 6 files under tests/ (67 tests), none import model-router or heartbeat today. Run with vitest under Node 22 (mise x node@22 -- npm test).
• Version banner: package.json, README header, src/cli.ts:676.
• Live persona ~/.mame/personas/Mame.yml currently names openrouter/x-ai/grok-4.20 — the class of id pi-ai's catalog rejects.

Implementation steps (ordered; each step compiles + tests green before the next)

Step 1 — New LLM client: src/llm/client.ts + src/llm/types.ts

src/llm/types.ts — Mame-owned wire types (OpenAI chat-completions shapes):

• LlmMessage: { role: "system" | "user" | "assistant" | "tool", content: string | ContentPart[], tool_calls?, tool_call_id? } with ContentPart = { type: "text", text } | { type: "image_url", image_url: { url } }. Both data: URLs and https URLs pass through verbatim in image_url.url — OpenRouter accepts both; delete the fetch-to-base64 conversion in agent.ts (fetchImageAsContent) since it's no longer needed.
• LlmTool: { type: "function", function: { name, description, parameters: <JSON Schema> } }.
• LlmResponse: parsed choices[0].message → { content: string | null, toolCalls: { id, name, arguments: Record<string, unknown> }[], finishReason }. arguments is JSON.parse'd defensively: on parse failure, keep the raw string in an error result rather than throwing.
• ThinkingLevel = "off" | "low" | "medium" | "high" (moved here from agent.ts; agent.ts re-exports or imports).

src/llm/client.ts:

• resolveRoute(model: string): { baseUrl, apiKey, modelId, label }
  • openrouter/<rest> → OpenRouter base URL + process.env.OPENROUTER_API_KEY, modelId <rest>. Missing key → clear error naming OPENROUTER_API_KEY.
  • else → MAME_LLM_BASE_URL/MAME_LLM_API_KEY, modelId = full string. Missing base URL → the "removed in 0.4.0" error above (must name openrouter as the path to anthropic/google models).
  • NO catalog, NO allowlist, NO validation of <rest>.
• chat(opts): Promise<LlmResponse> where opts = { model, system, messages, tools?, toolChoice?, thinkingLevel?, maxTokens? }.
  • POST {baseUrl}/chat/completions via fetch. Headers: Authorization: Bearer, Content-Type: application/json, plus OpenRouter attribution headers (HTTP-Referer/X-Title: Mame) when routing to openrouter (harmless elsewhere; only set for the openrouter route).
  • System prompt goes in as the first message with role "system".
  • thinkingLevel: "off" → omit; else include OpenRouter's reasoning param. Verify against current OpenRouter docs during implementation — as of the docs the shape is "reasoning": { "effort": "low" | "medium" | "high" }. Only attach it for the openrouter route; for the generic route attach it too (OpenAI-compatible endpoints ignore unknown fields or reject — if a provider rejects it, the verbatim error tells the user to set thinkingLevel off).
  • toolChoice: pass through; used by structured-output callers as { type: "function", function: { name: "submit_schedule" } }.
  • Retries: wrap the fetch in p-retry (retries: 2, minTimeout 1000, factor 2 — same budget as tools). Transient = network errors, HTTP 429/502/503/504 (reuse isTransientError — move it into src/llm/errors.ts or export from client and import in tools/index.ts so there is ONE definition). On 429, honor Retry-After (seconds or HTTP-date) when present and sane (< 60s) by using it as the wait; otherwise exponential backoff.
  • Non-2xx after retries: throw LlmError carrying status and the response body verbatim (truncate to ~2000 chars, strip HTML tags as the old code did for HTML error walls). Callers surface err.message to the agent/user — this is how "unknown model" now reads: the provider's own error text.

Update parseModelString in src/model-router.ts: it survives as the route parser (openrouter/... vs generic), but drop the google/anthropic backend values — return { backend: "openrouter" | "generic", modelId }. Alternatively fold it into resolveRoute and delete model-router.ts entirely (preferred: delete src/model-router.ts; move parseModelString into src/llm/client.ts and update the 5 importers: agent.ts, heartbeat.ts, onboard.ts, magazine/digest.ts, improve.ts).

Tests (new, tests/llm-client.test.ts, mock global fetch — no network):

• prefix routing: openrouter/x-ai/grok-4.5 → OpenRouter URL + model x-ai/grok-4.5; unprefixed model with MAME_LLM_BASE_URL set → generic route; unprefixed without it → error message mentions openrouter and MAME_LLM_BASE_URL.
• request shaping: system message first, tools serialized to OpenAI format, tool_choice passthrough, image_url parts preserved for data: and https: URLs.
• thinkingLevel mapping: off → no reasoning key; low/medium/high → reasoning.effort.
• error surfacing: 400 body text appears verbatim in thrown error; 429 then 200 → retried and succeeds; 4 consecutive 500s → throws after retry budget.
• tool_calls response parsing incl. malformed-JSON arguments.

Step 2 — Agent loop: src/llm/loop.ts

Owned replacement for pi-agent-core's Agent, shaped for Mame's three callers (think, heartbeat parse, onboarding):

• runAgentLoop(opts): Promise<LoopResult> with opts = { model, system, messages: LlmMessage[], tools: LoopTool[], thinkingLevel, toolChoice?, maxIterations? (default 25), maxTokens? } and LoopTool = { name, description, parameters: <JSON Schema>, execute(input) => Promise<unknown> }.
• Loop: call chat(); if the response has tool calls, execute each through the ported tool-execution path (Step 3) and append the assistant message + role:"tool" results, then iterate. Stop when the model returns no tool calls.
• Tool errors become role:"tool" results with { error, retriesExhausted } JSON (current behavior) — never crash the loop.
• toolChoice is applied only on the first iteration (forcing it every iteration would loop forever); after the forced call executes, the loop may stop (structured-output callers only need one iteration — they can pass maxIterations: 2).
• Iteration cap: on hitting maxIterations, log a warning (pino) and append/ return a final message noting the cap was hit, with whatever text the model last produced.
• Provider errors (LlmError) are NOT swallowed pi-style: runAgentLoop throws; each caller already has a try/catch that turns errors into friendly strings (agent.ts) or [] (heartbeat) or a send() (onboard). This changes the internal mechanism (state.errorMessage goes away) but preserves observable behavior: user sees "Something went wrong… Error: ".
• LoopResult = { messages: LlmMessage[], text: string /* last assistant text */, toolCallCount, hitIterationCap }.

Tests (new, tests/llm-loop.test.ts, mock chat):

• multi-turn: tool_calls → executes → feeds results → second response no tools → stops.
• tool execute() throwing → error tool result, loop continues, model sees it.
• iteration cap: model that always tool-calls stops at 25 with hitIterationCap.
• toolChoice forced on first request only.

Step 3 — Port tool execution + approval gate (src/tools/index.ts rewrite)

Collapse the two surfaces into one (this falls out naturally — both existing surfaces die with their consumers):

• Keep: registerTool, loadTools, ToolHandler (unchanged shape — definition.input_schema is already JSON Schema, which is exactly what the new client wants; individual tool files in src/tools/*.ts need zero changes), isTransientError (or re-export from llm/errors), enforceApproval verbatim — same AFFIRMATIVE_RE, same fail-closed semantics, same once-per-call placement outside the p-retry budget.
• New: getLoopTools(enabledTools: string[], turn: Turn): LoopTool[] — the p-retry + AbortError wrapper from today's toolHandlerToAgentTool, minus the pi types: enforceApproval first (once), then p-retry(execute) with AbortError for non-transient errors, result JSON.stringify'd. Thrown errors are caught by the loop (Step 2) and become error tool results.
• Delete: executeToolCalls, withRetry, getAgentTools, getToolDefinitions (unless improve/digest want raw definitions — they don't use tools), all @anthropic-ai/sdk and pi imports.

Tests (new or extended, tests/tools-approval.test.ts, mock askApproval):

• flagged tool + "yes" → executes once; flagged + "no" → error result containing the user's words, execute never called; askApproval throwing → fails closed.
• approval asked exactly once even when execute fails transiently and retries.

Step 4 — Migrate callers

• src/agent.ts: conversation buffer stores LlmMessage[]; build system prompt as today; images become image_url parts (pass URLs/data-URLs straight through, delete fetchImageAsContent); thinkingLevel resolution simplifies — no catalog means no piModel.reasoning flag, so: turn override > persona/ config default > "off" (document in a comment that reasoning-mandatory models now need an explicit thinkingLevel on the persona). Call runAgentLoop; persist result.messages to the buffer (heartbeat channel still bypasses); return result.text or the fallback string. Keep the outermost try/catch intact.
• src/heartbeat.ts: parseScheduleWithModel(model: string, markdown) now takes the model string (no more pre-resolved pi Model — the testability hook becomes "inject via mocked chat"). Same system prompt, same ScheduleSchema TypeBox object (now imported from @sinclair/typebox), passed as parameters on a single submit_schedule LoopTool whose execute captures entries (keep the empty-string→null project coercion, add explicit validation since AJV is gone: entries not matching the shape → log + []). Force it with toolChoice, maxIterations: 2. All failure modes still return [] and log. Delete the getModel resolution in parseSchedule. New tests (tests/heartbeat-parse.test.ts, mock the client): happy path (tool called with 2 entries → 2 HeartbeatEntry, "" project → null); model returns text without calling the tool → [] + logged; provider error → []; malformed entries → [].
• src/onboard.ts: keep the send/receive conversation structure; each user turn appends to a persistent LlmMessage[] and calls runAgentLoop with the onboarding tools (converted from AgentTool shape to LoopTool: same TypeBox parameter schemas, execute loses the _toolCallId first param and the { content, details } wrapper — return the message string). Errors → send() as today.
• src/improve.ts: replace chatCompletion with a single chat() call (no tools); conversation param type becomes the buffer's LlmMessage[]. Check its caller for the type change (grep maybeExtractSkill call sites).
• src/magazine/digest.ts: resolveModel returns the model string; llmJson/llmText call chat({ model, system, messages, maxTokens }) and read response.content. Behavior identical.
• Delete src/model-router.ts (parseModelString moved per Step 1).

Step 5 — Cleanup + docs

• package.json: remove @mariozechner/pi-ai, @mariozechner/pi-agent-core, @anthropic-ai/sdk; add @sinclair/typebox (pin to the version pi-ai was using to avoid schema-shape surprises). npm install to refresh lockfile.
• Grep for any straggler imports (mariozechner, @anthropic-ai/sdk) — must be zero.
• README:
  • Dependencies section: reflect removed/added packages.
  • Replace "20+ LLM backends" claims with the honest story: OpenRouter-first (any model OpenRouter serves, no client-side catalog — new models work the day they ship), plus any OpenAI-compatible endpoint via MAME_LLM_BASE_URL/MAME_LLM_API_KEY.
  • Model routing section: openrouter/<provider>/<model> format; note that direct google/* and anthropic/* ids were removed and show the openrouter/google/... / openrouter/anthropic/... replacements (README lines ~233, ~286–287 and any persona examples).
  • Note the thinkingLevel behavior (maps to OpenRouter reasoning.effort; reasoning-mandatory models need it set explicitly now).

Step 6 — Verification

All commands under Node 22: mise x node@22 -- <cmd>.

1. npx tsc --noEmit clean.
2. npm test — all pre-existing 67 tests pass unchanged, plus the new suites (llm-client, llm-loop, tools-approval, heartbeat-parse).
3. npm run build succeeds.
4. Live smoke test (vault has a real OPENROUTER_API_KEY; load it into env the same way src/cli.ts does): one-shot CLI run — mame chat (or a small script calling think()) with model openrouter/x-ai/grok-4.5:
   • confirm a text reply comes back;
   • confirm at least one tool call round-trips (prompt it to use memory or web_search);
   • confirm a bogus model id (openrouter/x-ai/grok-99) surfaces OpenRouter's error verbatim rather than a catalog message. Do NOT restart the running systemd mame service.

Step 7 — Version bump (final commit)

0.4.0 in package.json, README header, and the src/cli.ts banner (line ~676).

Commit plan

1. feat(llm): add owned OpenAI-compatible client (src/llm/) with tests
2. feat(llm): add owned agent loop with tool execution + approval gate ported (Steps 2–3 + their tests)
3. refactor: migrate agent/heartbeat/onboard/improve/digest off pi-ai (Step 4, incl. heartbeat parse tests; delete model-router.ts)
4. chore: drop pi-ai, pi-agent-core, anthropic sdk; update README (Step 5)
5. chore: bump version to 0.4.0 (Step 7, after Step 6 verification passes)

Risks / watchpoints

• OpenRouter reasoning param shape — verify against live docs before implementing; the plan assumes {"reasoning": {"effort": "..."}}.
• Reasoning-mandatory models (MiniMax M2.7, DeepSeek R1): the old code used the catalog's reasoning flag to auto-set "medium". Without a catalog these need thinkingLevel set on the persona; call this out in README. If the live Mame.yml heartbeat model breaks on this, set thinkingLevel in the persona docs.
• Message-history compatibility: the conversation buffer changes shape (pi Message[] → LlmMessage[]). Buffers are in-memory only (shutdown persists to memory as text), so no migration needed — verify getActiveConversations consumers in index.ts still compile.
• Tool-call/结果 adjacency: the old Gemini-via-pi 400 bug ("function response turn…") motivated heartbeat's buffer bypass — keep that bypass.
• onboard.ts inline tools: converting AgentTool→LoopTool must keep the send() progress callbacks working in both CLI and Signal paths.