feat: add local model provider for classifier routing (llama.cpp)

Support running the routing classifier on a local model (e.g. Gemma 3n 270M)
via llama.cpp server, eliminating external API dependency for routing decisions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: Alexey Panfilov

CLAUDE.md

@@ -48,6 +48,7 @@ flowchart LR
 - `openai_compat.go` — shared OpenAI-compatible implementation using raw `net/http` (no go-openai). `buildMessages(messages, systemPrompt, vision bool)` serialises messages with full control: assistant messages with `tool_calls` and empty content use `"content": null` (not omitted) to satisfy all provider APIs. `image_url` parts are replaced with `[image]` for non-vision providers; `input_audio` parts likewise. Defines `APIError{StatusCode, Message}` for fallback routing.
 - `ollama.go` — native Ollama provider using `/api/chat` protocol (not OpenAI-compat). Supports Ollama Cloud (`https://ollama.com`) and local instances. Handles tool calling (Ollama returns `arguments` as object → serialised to JSON string; generates `call_N` IDs since Ollama omits them). Multimodal: base64 images sent via `images[]` field (strips data URI prefix). `stream: false` for synchronous responses.
 - `claude_bridge.go` — Claude Bridge provider. Sends prompts to `claude-bridge` HTTP service on the host, which wraps `claude -p` CLI. `buildPrompt(messages, systemPrompt)` flattens multi-turn history into `User: ... / Assistant: ...` text format (CLI doesn't support multi-turn). Multimodal parts replaced with `[image]`/`[audio]`/`[document]` placeholders. Returns `APIError` on bridge errors for fallback routing. No tool calling — Claude CLI handles MCP tools itself via `--project-dir`.
+- `local.go` — local model provider via llama.cpp server (OpenAI-compatible API). No API key required — only `base_url` and `model`. Used for classifier routing with lightweight models like Gemma 3n 270M.
 - `router.go` — thread-safe routing config (`mu` protects both `override` and `cfg`). Priority: multimodal → override → classifier → reasoner → primary → fallback on 5xx/429/network. `SetRole(role, model)` + `SetClassifierMinLen(n)` allow runtime changes; `saveOverrides()` persists to `persistPath` (JSON) on every change; `LoadPersistedOverrides()` applies saved values on startup. Classifier has 5 s timeout; input truncated to 500 chars; logs routing decisions at Info when routed to reasoner.
 - All providers are optional in `main.go` — any configured model can be `routing.default`. The bot exits with a clear error if the default provider is missing.
 
@@ -190,7 +191,7 @@ Bot (Docker) → POST /ask → claude-bridge (host:9900) → claude -p → respo
 
 1. Implement `llm.Provider` interface (or reuse `openai_compat.go` if OpenAI-compatible)
 2. Add `ModelConfig` with `base_url` to `config.yaml` and `ModelsConfig` struct
-3. Wire in `main.go` with `if cfg.Models.X.APIKey != "" { addProvider("key", ...) }`
+3. Wire in `main.go` with `if cfg.Models.X.APIKey != "" { addProvider("key", ...) }` (for local models without API key, check `BaseURL` instead)
 4. Set as `routing.default` or a routing role
 
 ### Adding multimodal content types

README.md

@@ -4,7 +4,7 @@ A lightweight Telegram bot that acts as a personal AI assistant. Written in Go 
 
 ## Features
 
-- **Multi-model routing** — any configured model can be primary; automatic fallback on errors or rate limits; dedicated reasoner for complex tasks; vision model for images; classifier-based routing to reasoner
+- **Multi-model routing** — any configured model can be primary; automatic fallback on errors or rate limits; dedicated reasoner for complex tasks; vision model for images; classifier-based routing to reasoner (supports local models via llama.cpp server)
 - **Claude via bridge** — use Claude (Anthropic Max subscription) as a provider through a lightweight host-side bridge service that wraps `claude -p` CLI; no separate API key needed
 - **Ollama Cloud support** — native `/api/chat` provider for Ollama Cloud and local Ollama instances; tool calling, multimodal, Bearer auth
 - **Voice messages** — send a voice message in Telegram and it's automatically transcribed via the multimodal model (Gemini), then processed as text through the normal pipeline; replies include both text and a voice message via Edge TTS
@@ -213,12 +213,20 @@ models:
   #   api_key: ${CLAUDE_BRIDGE_TOKEN}
   #   max_tokens: 120              # timeout in seconds
 
+  # Local model via llama.cpp server (OpenAI-compatible API).
+  # No API key needed — only base_url and model.
+  # local:
+  #   provider: local
+  #   model: gemma-3n-E2B
+  #   max_tokens: 64
+  #   base_url: http://classifier:8080/v1
+
 routing:
   default: deepseek          # primary model — can be any configured model name
   fallback: gemini-flash     # also used for multimodal
   multimodal: gemini-flash
   reasoner: deepseek-r1
-  classifier: deepseek       # model for reasoning detection; omit to disable
+  classifier: local          # local model for reasoning detection; omit to disable
   classifier_min_length: 100 # min chars to run classifier; 0 = disabled
   compaction_model: deepseek
 
@@ -372,7 +380,7 @@ esphome run atom-echo.yaml            # compile + flash via USB
 | 3 | `primary` | Default for all other messages |
 | 4 | `fallback` | Primary unavailable (5xx / 429 / network error) |
 
-The classifier is a lightweight call with no history and no tools that returns `yes`/`no`. It only runs for messages longer than `classifier_min_length` characters (default: 100). Input is truncated to 500 chars to save tokens. Set `classifier_min_length: 0` to disable.
+The classifier is a lightweight call with no history and no tools that returns `yes`/`no`. It only runs for messages longer than `classifier_min_length` characters (default: 100). Input is truncated to 500 chars to save tokens. Set `classifier_min_length: 0` to disable. The classifier can run on a local model (e.g. Gemma 3n 270M via llama.cpp server) for zero-latency, zero-cost routing — configure `local` provider and set `classifier: local`.
 
 All routing roles can be changed live via `/routing` — an inline keyboard menu. **Changes persist across restarts** in `config/routing.json`. On startup, the bot notifies the owner via Telegram if any routing role references an unavailable model.
 

cmd/agent/main.go

@@ -92,6 +92,10 @@ func main() {
 		p, e := llm.NewClaudeBridge(cfg.Models.Claude)
 		addProvider("claude", p, e)
 	}
+	if cfg.Models.Local.BaseURL != "" {
+		p, e := llm.NewLocal(cfg.Models.Local)
+		addProvider("local", p, e)
+	}
 
 	// Ensure the default routing provider is available.
 	if providers[cfg.Routing.Default] == nil {

config/config.yaml

@@ -69,6 +69,14 @@ models:
     max_tokens: 8192
     base_url: https://dashscope-intl.aliyuncs.com/compatible-mode/v1
 
+  # Local model via llama.cpp server (OpenAI-compatible API).
+  # Used as classifier for routing decisions. No API key needed.
+  local:
+    provider: local
+    model: gemma-3n-E2B
+    max_tokens: 64
+    base_url: http://classifier:8080/v1
+
   # Claude via bridge (host service wrapping `claude -p` CLI).
   # Requires claude-bridge running on host:9900.
   # max_tokens here is used as timeout_sec for the CLI call.
@@ -83,7 +91,7 @@ routing:
   multimodal: gemini-flash
   reasoner: claude
   compaction_model: deepseek
-  classifier: qwen-flash
+  classifier: local
   classifier_min_length: 100
 
 tool_filter:

internal/config/config.go

@@ -82,6 +82,7 @@ type ModelsConfig struct {
 	QwenMax         ModelConfig `yaml:"qwen-max"`
 	Ollama          ModelConfig `yaml:"ollama"`
 	Claude          ModelConfig `yaml:"claude"`
+	Local           ModelConfig `yaml:"local"`
 }
 
 

internal/llm/local.go

@@ -0,0 +1,9 @@
+package llm
+
+import "telegram-agent/internal/config"
+
+// NewLocal creates an OpenAI-compatible provider for a local model (e.g. llama.cpp server).
+// Does not require an API key — only base_url and model are needed.
+func NewLocal(cfg config.ModelConfig) (*openAICompatProvider, error) {
+	return newOpenAICompat(cfg, "", "local")
+}

internal/llm/openai_compat.go

@@ -33,8 +33,8 @@ type openAICompatProvider struct {
 }
 
 func newOpenAICompat(cfg config.ModelConfig, defaultBaseURL, providerName string, vision ...bool) (*openAICompatProvider, error) {
-	if cfg.APIKey == "" {
-		return nil, fmt.Errorf("%s: api_key is required", providerName)
+	if cfg.APIKey == "" && cfg.BaseURL == "" {
+		return nil, fmt.Errorf("%s: api_key or base_url is required", providerName)
 	}
 	baseURL := cfg.BaseURL
 	if baseURL == "" {