Skip to content

Commit 860736e

Browse files
author
MLuqmanBR
committed
feat(providers): custom providers as first-class platform objects (tashfeenahmed#230)
Redesign custom providers from a per-model {baseUrl, modelId, apiKey} blob into first-class platform objects that work like the built-in ones. The operator adds a provider once (slug + display name + base URL), then adds as many models as they like under it. Multiple API keys for the same custom provider now round- robin across a single shared base URL. Server: - server/src/db/index.ts — new custom_providers table (id, slug UNIQUE, display_name, base_url, created_at) + V23 migration that re-points any existing 'custom' rows to a synthetic 'legacy-custom' provider row, so no data is lost on upgrade. - shared/types.ts — Platform no longer includes 'custom'. New CustomProvider/CustomProviderCreate/CustomProviderUpdate/CustomModel* types. Model.platform and ApiKey.platform are relaxed to string so they can hold any custom slug. - server/src/providers/index.ts — add buildProviderFor(platformSlug): returns the registered singleton for built-ins, or constructs an OpenAICompatProvider bound to the custom_providers row's base_url for custom slugs. The old resolveProvider(platform, baseUrl) overload is gone. - server/src/routes/custom.ts — new, 8 endpoints: GET /api/custom-providers POST /api/custom-providers PATCH /api/custom-providers/:slug DELETE /api/custom-providers/:slug (cascades to models, api_keys, fallback_config) GET /api/custom-providers/:slug/models POST /api/custom-providers/:slug/models PATCH /api/custom-models/:id DELETE /api/custom-models/:id Slug regex ^[a-z0-9][a-z0-9-]{1,30}[a-z0-9]$ (2-32 chars, no leading/trailing dash). Built-in slugs (google/groq/...) are deny-listed. PATCH on custom_providers keeps api_keys.base_url in lockstep. - server/src/routes/keys.ts — drop POST /api/keys/custom. POST /api/keys now validates platform via buildProviderFor and stores base_url on the key row for custom slugs. PATCH /api/keys/platform/:platform now accepts custom slugs (verified against custom_providers). - server/src/services/router.ts — uses buildProviderFor(entry.platform). Drop the 'custom && key_id != null' branch; the models.key_id column is now a denormalized legacy column that the router doesn't consult. - server/src/services/health.ts — uses buildProviderFor(row.platform). - server/src/app.ts — mount customRouter behind a path-aware requireAuth guard so /api/ping and /api/auth/* stay public. The guard regex-scopes requireAuth to /api/custom-providers/* and /api/custom-models/* only. Tests: - server/src/__tests__/routes/custom-providers.test.ts — 16 tests across provider CRUD, model CRUD, cascade deletes, slug-collision rejection, buildProviderFor integration. - server/src/__tests__/routes/proxy-empty-completion.test.ts, server/src/__tests__/routes/responses-tool-args-repair.test.ts — add buildProviderFor: () => fakeProvider to the vi.mock factory for the new provider-lookup function. Client: - client/src/pages/KeysPage.tsx — full rewrite. New components: AddPlatformModal, EditPlatformModal, PlatformTile, PlatformsSection, CustomModelsSection. Custom provider tiles get Edit/Remove; the 'Add New Platform' tile is always last in the provider list. Docs: - README.md — update the custom-provider blurb and add a 'Custom platforms and models' subsection covering slug rules, modal fields, model form essential-vs-advanced toggle, and cascade-on-delete behavior. Behavior change worth highlighting: the 'model belongs to one specific key' binding (the models.key_id column) is intentionally dropped. Any enabled key for the custom provider can now serve any model on it.
1 parent d42380b commit 860736e

14 files changed

Lines changed: 1477 additions & 570 deletions

File tree

README.md

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -69,7 +69,21 @@ The problem is that stacking them by hand is painful: sixteen different SDKs, si
6969
</tr>
7070
</table>
7171

72-
Plus a **custom** provider — point at any OpenAI-compatible endpoint (llama.cpp, LM Studio, vLLM, a local Ollama, or a remote gateway) from the Keys page.
72+
Plus **your own custom platforms and models** — add any OpenAI-compatible endpoint (llama.cpp, LM Studio, vLLM, a local Ollama, or a remote gateway) as a first-class provider from the Keys page, then add individual models under it. They route through the same fallback chain, score against the same bandit, and show up alongside built-in platforms everywhere.
73+
74+
### Custom platforms and models
75+
76+
From the Keys page, the **Platforms** grid is the unified catalog view. Every built-in platform you've added a key for shows up alongside every custom platform you've registered. The grid ends with an **Add New Platform** tile that opens a modal for:
77+
78+
- **Slug** — a short identifier like `my-ollama` (lowercase letters, digits, dashes; 2-32 chars; cannot collide with a built-in).
79+
- **Display name** — shown in the dashboard.
80+
- **Base URL** — the OpenAI-compatible endpoint, e.g. `http://192.168.1.10:11434/v1`.
81+
82+
Once a platform exists, the **Add a model** form below it lets you register any number of individual models on that platform. Only the essentials are required (model id, display name, context window, supports-tools, supports-vision); an **Advanced** toggle exposes intelligence rank, speed rank, size label, and the four rate-limit counters for users who care. The model joins the fallback chain at the lowest priority and shows up everywhere built-in models do — `/v1/models`, the Fallback page, the Analytics page.
83+
84+
Adding an API key for a custom platform works the same as for a built-in: pick the custom slug in the **Add a provider key** form, paste the bearer (or leave blank for local servers that don't need one), and the key routes to your endpoint.
85+
86+
Removing a custom platform cascades — it drops every model on that platform, every key, and every fallback entry. There's no "leaving a model orphaned" state.
7387

7488
## Features
7589

@@ -83,7 +97,7 @@ Plus a **custom** provider — point at any OpenAI-compatible endpoint (llama.cp
8397
- **Sticky sessions** — Multi-turn conversations keep talking to the same model for 30 minutes to avoid the hallucination spike that comes from mid-conversation model switches.
8498
- **Encrypted key storage** — API keys are encrypted with AES-256-GCM before hitting SQLite; decryption happens in-memory just before a request.
8599
- **Unified API key** — Clients authenticate to your proxy with a single `freellmapi-…` bearer token. You never expose upstream provider keys to your apps.
86-
- **Dashboard login**The admin UI and all `/api/*` routes are gated behind an email + password account (scrypt-hashed, session-token auth), set on first run. The `/v1` proxy keeps its own unified-key auth for apps.
100+
- **LAN auto-trust**FreeLLMAPI is a single-user tool, so the admin UI and `/api/*` routes skip the login form whenever the request comes from the local machine (loopback, RFC1918, link-local, IPv6 ULA / link-local). Remote callers still need an email + password account (scrypt-hashed, session-token auth), set on first run. The `/v1` proxy keeps its own unified-key auth for apps.
87101
- **Health checks** — Periodic probes mark keys as `healthy`, `rate_limited`, `invalid`, or `error` so the router skips dead ones automatically.
88102
- **Admin dashboard** — React + Vite UI to manage keys, reorder the fallback chain, inspect analytics, and run prompts in a playground. Dark mode included.
89103
- **Analytics** — Per-request logging with latency, token counts, success rate, and per-provider breakdowns.

0 commit comments

Comments
 (0)