Skip to content

feat: ✨ LiteLLM multi-provider model selection + catalog-driven UI - #111

Merged
nj-io merged 1 commit into
developfrom
feat/model-selection-litellm
Jul 8, 2026
Merged

feat: ✨ LiteLLM multi-provider model selection + catalog-driven UI#111
nj-io merged 1 commit into
developfrom
feat/model-selection-litellm

Conversation

@nj-io

@nj-io nj-io commented Jul 8, 2026

Copy link
Copy Markdown
Owner

What

Replaces the hand-rolled OpenAICompatClient with a LiteLLM-backed LiteLLMClient (openai/openrouter/ollama) behind the unchanged LLMClient.complete() interface, and makes the model catalog the single source of truth wired to the settings UI. Goal: select OpenRouter models like openrouter/z-ai/glm-5.2 reliably, with real per-call cost tracking.

Adopts the proven pattern from mirra's PromptLab (LiteLLM + registry-as-cost-truth), adapted to this project's tool-calling contract.

Key changes

  • llm/litellm_client.py (new): forces the single expected tool; on a tool-less reply, parses JSON from the message body, else returns a text block (never raises) so the Gatekeeper's prose fallback is preserved; real cost via LiteLLM response_cost → catalog fallback → honest unknown, with a cost_source field.
  • catalog.py: single source of cost truth (estimate_cost_cents, get_all_models, get_model_by_full_id); refreshed to current models (Opus 4.8 / Sonnet 5 / Haiku 4.5 / Fable 5) + verified OpenRouter models incl. z-ai/glm-5.2.
  • client.py (Anthropic, same transport) now sources cost from the catalog — fixes silent $0 cost for current model IDs. claude-cli reports cost_source="subscription" ($0), isolated for easy change if subscription calls ever bill.
  • parsing.extract_json_object: shared boundary JSON parser (promoted from a private claude_cli helper that dragged a Unix-only pwd import into the OpenRouter path).
  • Web: GET /api/models endpoint + OpenRouter key validation; the settings Models section is now catalog-driven (provider-grouped dropdown showing cost/context) instead of three drifting hardcoded lists. Free-text provider/model-id retained.
  • ClaudeClient / ClaudeCliClient untouched (Anthropic SDK + claude -p subscription path stay as-is); litellm is an optional dep (multi-provider extra, pinned >=1.91.0).

Verification

  • ruff + mypy clean; test suite green (new test_litellm_client.py; updated catalog/factory/llm_client/claude_cli; test_openai_compat_client.py removed).
  • Live openrouter/z-ai/glm-5.2 call through the factory: real tool call + cost 0.0505¢, source=provider.
  • Browser UI round-trip: catalog dropdown → select → auto-save → persisted to config.yaml.

Notes

  • Plan-reviewed before implementation (an adversarial review caught the Anthropic $0-cost bug, the pwd import leak, and a Gatekeeper regression — all fixed here).
  • Branched from clean develop HEAD, so it is isolated from unrelated in-progress media/buttons work in the local tree.
  • Doc updates (REUSABILITY.md, docs/CLAUDE.md) live in the separate docs repo and are committed there, not in this PR.

🤖 Generated with Claude Code

Replace the hand-rolled OpenAICompatClient with a LiteLLM-backed
LiteLLMClient for openai/openrouter/ollama, behind the unchanged
LLMClient.complete() interface. Makes OpenRouter models like
openrouter/z-ai/glm-5.2 reliably selectable.

- llm/litellm_client.py: forces the single expected tool; falls back to
  JSON-from-text and returns a text block (never raises) so the Gatekeeper
  prose fallback is preserved; real per-call cost via LiteLLM with a
  catalog fallback and cost_source tracking.
- catalog.py: single source of cost truth (estimate_cost_cents,
  get_all_models, get_model_by_full_id); refreshed to current models
  (Opus 4.8 / Sonnet 5 / Haiku 4.5 / Fable 5) + verified OpenRouter
  models incl. z-ai/glm-5.2.
- client.py (Anthropic) sources cost from the catalog (fixes $0 cost for
  new model IDs); claude-cli reports cost_source="subscription" ($0).
- parsing.extract_json_object: shared boundary JSON parser (promoted from
  a private claude_cli helper that pulled in a Unix-only import).
- web: GET /api/models endpoint + OpenRouter key validation; the settings
  Models section is now catalog-driven (provider-grouped dropdown with
  cost/context) instead of a hardcoded preset list.
- litellm added as an optional dep (multi-provider extra, pinned >=1.91.0).

Verified: ruff + mypy clean, tests green, live openrouter/z-ai/glm-5.2
call (real tool-call + cost source=provider), and a browser UI round-trip
that persists the selection to config.yaml.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@nj-io
nj-io merged commit 9ef757f into develop Jul 8, 2026
1 of 6 checks passed
@nj-io
nj-io deleted the feat/model-selection-litellm branch July 9, 2026 02:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant