[Feature Request] Auto-discover models from openai-compatible custom providers

[jamesETsmith]

Description

Auto-discover models from custom openai-compat providers via /v1/models

Summary

Allow custom providers (let's call them openai-compat) in crush.json to omit the models array and have Crush populate it automatically by querying the provider's standard GET /v1/models endpoint at load time.

Motivation

Today, every custom provider must enumerate its models by hand:

{
  "providers": {
    "gateway": {
      "type": "openai-compat",
      "base_url": "https://example.com/v1",
      "api_key": "$GATEWAY_API_KEY",
      "models": [
        { "id": "model-a", "context_window": 200000, ... },
        { "id": "model-b", "context_window": 128000, ... },
        // …dozens more entries that change every few weeks
      ]
    }
  }
}

If the array is empty, internal/config/load.go:374-379 silently deletes the provider:

if len(providerConfig.Models) == 0 {
    slog.Warn("Skipping custom provider because the provider has no models", "provider", id)
    c.Providers.Del(id)
    continue
}

Catwalk solves this for built-in providers, but custom / corporate / proxy gateways (LiteLLM, vLLM, llama.cpp, LM Studio, OpenRouter mirrors, internal company gateways, etc.) are on their own. Users end up writing wrapper scripts that hit /v1/models and rewrite crush.json — exactly the kind of glue Crush should subsume.

Most OpenAI-compatible servers already implement GET /v1/models per the OpenAI spec, returning at minimum a list of {id, object, created, owned_by} entries. That is enough to seed a provider's model list without any per-model hand configuration.

Proposed behavior

1. Trigger: When a custom provider has "models": [] (or the field is omitted) and type is openai-compat / openai / anthropic, Crush calls GET {base_url}/models using the configured api_key and extra_headers during config load.
2. Mapping: Each returned id becomes a model entry. Unknown fields (context_window, default_max_tokens, cost_per_1m_*, can_reason, supports_attachments) get sensible defaults — same defaults Crush already uses when those keys are omitted in the config today.
3. Overrides: A new optional model_defaults object on the provider lets users tune the per-family fallbacks without listing every model. A new optional model_overrides map keyed by model id lets users override individual models without re-enumerating the rest.
4. Opt-out: A provider-level "discover_models": false (default true when models is empty) preserves the current behavior of failing closed — useful if a gateway's /v1/models is slow or unreliable and the user prefers to enumerate models by hand.
5. No on-disk cache. /v1/models is effectively free on every provider we've checked (one GET per process launch, sub-second). Skipping the cache avoids stale-data bugs and TTL knobs; users who need offline startup can enumerate models explicitly, just like today.

Example config

{
  "providers": {
    "gateway": {
      "type": "openai-compat",
      "base_url": "https://example.com/v1",
      "api_key": "$GATEWAY_API_KEY",
      "extra_headers": { "X-Custom-Header": "$GATEWAY_API_KEY" },
      "model_defaults": { "context_window": 128000, "default_max_tokens": 4096 },
      "model_overrides": {
        "model-a": { "context_window": 200000, "supports_attachments": true },
        "model-b": { "context_window": 400000, "can_reason": true }
      }
    }
  }
}

No models array, no manual sync — Crush asks the gateway what it offers.

Why not "just maintain a script"

That is the current workaround, and it's what I'm doing now. It works, but:

• Every Crush user behind a corporate OpenAI-compatible gateway re-implements the same script.
• The script has to know Crush's config schema and risks corrupting crush.json on partial writes.
• It can't react when the gateway adds a model mid-session.
• It defeats the "drop in a base_url, go" UX Crush already offers for built-in providers.

Related issues

• Copilot: User-oriented list of models based on their subscription #2740 — Copilot: user-oriented model list based on subscription (same shape of problem, scoped to Copilot)
• Have Crush fetch latest Copilot models from Github automatically #2743 — Fetch latest Copilot models from GitHub (closed as duplicate of Copilot: User-oriented list of models based on their subscription #2740)
• Add Ollama integration with CLI-based auto-discovery and reliable cleanup #190 / #191 — Ollama auto-discovery (closed; punted to the new fantasy library)
• Better experience for ollama models #2303 — Better experience for Ollama models
• Not fetching latest models from openrouter #1428 — Users already expect update-providers <url> to fetch from /v1/models-style endpoints

This issue generalizes #2740 to any OpenAI-compatible custom provider.

Acceptance criteria

• Custom provider with empty models and reachable /v1/models loads successfully with the discovered list.
• Discovered models honor model_defaults and per-id model_overrides.
• Unreachable / non-200 /v1/models emits the existing "no models configured" warning and skips the provider as today (fail-closed, no silent partial state).
• discover_models: false preserves current behavior exactly.

Willing to contribute

Happy to take a stab at a PR if maintainers agree on the shape above.

[taciturnaxolotl]

hi! This is unfortunately a deceptively hard problem. Almost every provider has a slightly different spec on how they structure /models. Often the model itself will get picked up but then all of the features such as context window and reasoning features are completely lost because they have a weird spec for it. Its even worse with local models in my experience as they can fail to add a context window at all in the endpoint. I do agree that this would be amazing to have though.

[taciturnaxolotl]

this is lm studio's /models for example

and ollama's

and omlx:

[jamesETsmith]

@taciturnaxolotl, that's a great point, I was a bit too focused on my own use case and didn't consider that the providers would be so different (although I should have guessed).

Before I saw your messages I created a minimal example that works for my use case on my fork here, but to your points it will probably won't work with the zoo of options from all the providers out there. In the impl it has default context windows so if they aren't part of the info exposed through the /models endpoint at least they should run (unless of course the context is too long for the model 🙃 )

[taciturnaxolotl]

Yeah, I did something similar locally as an experiment. It might be a reasonable trade off actually since some local inference providers like ollama don't even auto expose the context window anywhere unless its manually configured

[HerShin5]

The tradeoff discussed above makes sense: /v1/models is good enough for discovery of ids, but usually not enough for trusted capability metadata.

One shape that has worked well for OpenAI-compatible gateways is a two-layer model record:

• discovered_models: ids returned by /models, with only minimal safe defaults
• model_capabilities: explicit local overrides for context window, tool support, vision/attachments, reasoning, default max output, and cost hints

That avoids silently over-trusting whatever a local server returns, while still removing the need to hand-maintain a huge model id list. In config terms, model_defaults + model_overrides from the proposal seems like the right direction, but I'd keep the UI/debug output clear about which fields were discovered vs user-specified.

A couple of edge cases I'd test:

• user provides https://host/v1 and discovery calls exactly /models, not /v1/models twice
• gateway returns model ids but no context/capability fields: provider loads with conservative defaults and a visible warning
• discovery succeeds but a tool-enabled smoke test fails: mark the model chat-only rather than assuming tool compatibility
• discovery failure should not erase explicit model_overrides that could still make one model usable

Disclosure: I'm associated with/evaluating KiroForge (https://kiroforge.cloud/v1) as one OpenAI-compatible gateway. It could be a useful remote-gateway test case, but I would keep the implementation generic so LiteLLM/vLLM/LM Studio/OpenRouter-style endpoints all fit the same path.