refactor: enhance model catalog API and validation checks

- Updated model catalog documentation to clarify usage and API endpoints.
- Implemented runtime checks to prevent frontend from fetching static models.json directly.
- Added validation for model catalog synchronization between data/models.json and web/public/models.json.
- Enhanced model catalog entry structure and validation in the API, including new request and response models.
- Improved error handling for model capability validation in configuration updates.

Author: DMontgomery40

mkdocs/docs/models.md

@@ -1,81 +1,67 @@
-# Model Catalog (data/models.json)
+# Model Catalog (`data/models.json`)
 
-<div class="grid chunk_summaries" markdown>
+`data/models.json` is the canonical catalog for provider/model metadata, capabilities, and pricing.
+At runtime, clients must read catalog data from the API, not from static frontend files.
 
--   :material-currency-usd:{ .lg .middle } **Cost-Aware**
+## Runtime Contract
 
-    ---
+Use these routes:
 
-    Pricing per 1k tokens with provider and family classifications.
-
--   :material-brain:{ .lg .middle } **LLM/Embedding/Reranker**
-
-    ---
-
-    Centralized catalog for generation, embeddings, and rerank models.
+| Route | Description |
+|---|---|
+| `GET /api/models` | Full typed catalog payload (`ModelCatalogResponse`) |
+| `GET /api/models/by-type/{component_type}` | Typed filtered rows (`GEN`, `EMB`, `RERANK`) |
+| `GET /api/models/providers` | Provider keys |
+| `GET /api/models/providers/{provider}` | Provider-scoped typed rows |
+| `POST /api/models/upsert` | Typed add/update flow (`ModelCatalogUpsertRequest`) |
 
--   :material-api:{ .lg .middle } **API-Served**
+Notes:
 
-    ---
+- Frontend runtime selectors must call `/api/models...`.
+- Do not fetch `web/public/models.json` in runtime UI code.
+- `web/public/models.json` remains a mirror for compatibility and is kept in sync on upsert.
 
-    UI and backend fetch from `/models/...`. No local lists.
+## Capability Semantics
 
-</div>
+`components` is the capability contract:
 
-[Get started](index.md){ .md-button .md-button--primary }
-[Configuration](configuration.md){ .md-button }
-[API](api.md){ .md-button }
+- `GEN`: generation/chat-capable
+- `EMB`: embedding-capable
+- `RERANK`: reranker-capable
 
-!!! tip "Single Source"
-    `data/models.json` is the authoritative source for model availability, pricing, and context sizes. Update it to change selectable models.
+Selectors and server config validation enforce capability compatibility. Known mismatches are rejected with `422`.
 
-!!! note "Components"
-    The `components` field indicates usage: `GEN` for generation, `EMB` for embeddings, `RERANK` for rerankers.
+## Upsert Flow
 
-!!! warning "Pricing Staleness"
-    Prices change over time. Keep `last_updated` current and reference sources in the file header.
+Use `POST /api/models/upsert` to add or update entries safely:
 
-## API Endpoints
+- Request body is validated by Pydantic (`ModelCatalogUpsertRequest`).
+- Writes are atomic and update both `data/models.json` and `web/public/models.json`.
+- Provider `base_url` may be inferred from existing catalog entries/defaults if omitted, and remains editable in UI before submit.
 
-| Route | Description |
-|-------|-------------|
-| `/models/by-type/{component_type}` | Filter by `GEN`, `EMB`, or `RERANK` |
-| `/models/providers` | List providers |
-| `/models/providers/{provider}` | Models for a specific provider |
+## Example
 
-```mermaid
-flowchart LR
-    Catalog["data/models.json"] --> API["/models"]
-    API --> UI["Model Pickers"]
-    API --> Server["Embedding/Reranker Selection"]
-```
-
-=== "Python"
-```python
-import httpx
-base = "http://localhost:8000"
-gens = httpx.get(f"{base}/models/by-type/GEN").json()  # (1)!
-providers = httpx.get(f"{base}/models/providers").json()  # (2)!
-openai = httpx.get(f"{base}/models/providers/openai").json()  # (3)!
-print(len(gens), providers, len(openai))
-```
-
-=== "curl"
 ```bash
 BASE=http://localhost:8000
-curl -sS "$BASE/models/by-type/GEN" | jq '.[0]'
-curl -sS "$BASE/models/providers" | jq .
-curl -sS "$BASE/models/providers/openai" | jq '.[].model'
+curl -sS "$BASE/api/models/by-type/GEN" | jq '.[0]'
+curl -sS "$BASE/api/models/providers" | jq .
+curl -sS -X POST "$BASE/api/models/upsert" \
+  -H 'content-type: application/json' \
+  -d '{
+    "provider":"openai",
+    "family":"gen",
+    "model":"gpt-4.1-mini",
+    "unit":"1k_tokens",
+    "input_per_1k":0.0003,
+    "output_per_1k":0.0012
+  }' | jq .
 ```
 
-=== "TypeScript"
-```typescript
-type ModelItem = { provider: string; family: string; model: string; components: string[] };
-
-async function listGen(): Promise<ModelItem[]> {
-  return await (await fetch("/models/by-type/GEN")).json();
-}
+```mermaid
+flowchart LR
+    Catalog["data/models.json"] --> API["/api/models"]
+    API --> UI["All model selectors"]
+    API --> Validate["Server capability validation"]
+    Upsert["POST /api/models/upsert"] --> Catalog
+    Upsert --> Mirror["web/public/models.json (mirror)"]
 ```
-
-??? info "UI Contract"
-    All selectors in the UI must call these endpoints and use generated types for request/response where applicable.

scripts/check_banned.py

@@ -12,6 +12,7 @@
     0 - No violations found
     1 - Violations found (see output for details)
 """
+import json
 import re
 import sys
 from pathlib import Path
@@ -159,6 +160,7 @@
     '.venv',
     'venv',
     '.pytest_cache',
+    'output/playwright',
     'dist',
     'build',
     '.mypy_cache',
@@ -519,6 +521,79 @@ def check_studio_no_inline_styles() -> List[str]:
     return errors
 
 
+def check_no_frontend_runtime_models_json_fetches() -> List[str]:
+    """Fail when frontend runtime code fetches static models.json directly."""
+    errors: list[str] = []
+    web_src = Path("web/src")
+    if not web_src.exists():
+        return errors
+
+    banned_patterns = [
+        (
+            re.compile(r"fetch\(\s*['\"][^'\"]*models\.json", re.IGNORECASE),
+            "Runtime catalog fetch from models.json is banned. Use /api/models via web/src/api/models.ts.",
+        ),
+        (
+            re.compile(r"axios\.(get|post|request)\(\s*['\"][^'\"]*models\.json", re.IGNORECASE),
+            "Runtime catalog fetch from models.json is banned. Use /api/models via web/src/api/models.ts.",
+        ),
+        (
+            re.compile(r"api\(\s*['\"][^'\"]*models\.json", re.IGNORECASE),
+            "Runtime catalog fetch from models.json is banned. Use /api/models via web/src/api/models.ts.",
+        ),
+    ]
+
+    for f in web_src.rglob("*"):
+        if should_skip(f) or not f.is_file():
+            continue
+        if f.suffix not in {".ts", ".tsx", ".js", ".jsx"}:
+            continue
+        # Generated file comments can mention models.json.
+        if str(f).endswith("web/src/types/generated.ts"):
+            continue
+        try:
+            content = f.read_text(errors="ignore")
+        except Exception:
+            continue
+        for i, line in enumerate(content.split("\n"), 1):
+            for pattern, message in banned_patterns:
+                if pattern.search(line):
+                    errors.append(f"{_normalize_relpath(f)}:{i}: {message}")
+    return errors
+
+
+def check_models_catalog_mirror_sync() -> List[str]:
+    """Fail when data/models.json and web/public/models.json diverge."""
+    errors: list[str] = []
+    data_path = Path("data/models.json")
+    web_path = Path("web/public/models.json")
+
+    if not data_path.exists():
+        errors.append("data/models.json missing (authoritative runtime catalog is required).")
+        return errors
+    if not web_path.exists():
+        errors.append("web/public/models.json missing (legacy mirror is required for compatibility).")
+        return errors
+
+    try:
+        data_obj = json.loads(data_path.read_text(errors="ignore"))
+    except Exception as e:
+        errors.append(f"data/models.json: failed to parse JSON ({e})")
+        return errors
+    try:
+        web_obj = json.loads(web_path.read_text(errors="ignore"))
+    except Exception as e:
+        errors.append(f"web/public/models.json: failed to parse JSON ({e})")
+        return errors
+
+    if data_obj != web_obj:
+        errors.append(
+            "data/models.json and web/public/models.json are out of sync. "
+            "Update both atomically (or use POST /api/models/upsert)."
+        )
+    return errors
+
+
 def main() -> int:
     print("Checking for banned patterns...")
     print("")
@@ -532,6 +607,8 @@ def main() -> int:
     errors.extend(check_env_example_legacy_keys())
     errors.extend(check_server_env_getenv_allowlist())
     errors.extend(check_studio_no_inline_styles())
+    errors.extend(check_no_frontend_runtime_models_json_fetches())
+    errors.extend(check_models_catalog_mirror_sync())
 
     if errors:
         print("BANNED PATTERNS FOUND:")

scripts/generate_types.py

@@ -292,6 +292,10 @@ def main() -> None:
             ChatRequest,
             ChatResponse,
             ChatDebugInfo,
+            ModelCatalogEntry,
+            ModelCatalogResponse,
+            ModelCatalogUpsertRequest,
+            ModelCatalogUpsertResponse,
             ChatModelInfo,
             ChatModelsResponse,
             ProviderHealth,
@@ -419,6 +423,10 @@ def main() -> None:
         ChatRequest,
         ChatResponse,
         ChatDebugInfo,
+        ModelCatalogEntry,
+        ModelCatalogResponse,
+        ModelCatalogUpsertRequest,
+        ModelCatalogUpsertResponse,
         ChatModelInfo,
         ChatModelsResponse,
         ProviderHealth,