Skip to content

Default /v1/models to downloaded-only, leave /models unchanged - #2247

Open
alytaphoenix wants to merge 1 commit into
exo-explore:mainfrom
alytaphoenix:feature/v1-models-downloaded-only
Open

Default /v1/models to downloaded-only, leave /models unchanged#2247
alytaphoenix wants to merge 1 commit into
exo-explore:mainfrom
alytaphoenix:feature/v1-models-downloaded-only

Conversation

@alytaphoenix

Copy link
Copy Markdown

Summary

/v1/models (the OpenAI-compatible model list, used by external clients/agents/integrations) was advertising every model card exo knows about, including ones nobody on the cluster has downloaded yet. Agents/tools that discover available models via this endpoint would see — and could try to call — models that don't exist locally.

/models (the dashboard's own model browser, used for finding new models to download) still needs to list everything by default. Both routes shared a single get_models handler, and a query-param default can't diverge per route, so this adds a thin get_v1_models wrapper registered only on /v1/models that defaults status="downloaded" and delegates to the existing (already-shipped) status=downloaded filter. /models keeps its unchanged default. ?status=all still returns the full list from either route — nothing that needs the full list is stranded.

Known limitations

  • "Downloaded" is a loose proxy for "usable," not a guarantee. A model downloaded to disk but not yet placed as a running instance still 404s ("No instance found for model X") on first use — see API._validate_model_has_instance. This change only stops agents from being offered models that don't exist on the cluster at all; it doesn't guarantee a listed model is actually ready to serve on the first request.
  • A multi-node model shows as "downloaded" once any single node has completed its download — not necessarily every node needed to actually serve it.

Out of scope (separate follow-up)

This is a default-filter flip only. A more granular "advertise this specific downloaded model to agents" checkbox — letting users curate a subset of downloaded models rather than all-or-nothing — would need persisted, cluster-synced state (a new command + event + State field + apply() logic) plus dashboard UI. That's a categorically bigger change than this one and is left as its own follow-up PR; this filter is forward-compatible with it (later: advertised = downloaded AND checked).

Test plan

  • uv run basedpyright on touched files — 0 errors; full-repo baseline unchanged (319 errors, 110 warnings)
  • uv run ruff check / ruff format — pass
  • New tests: /v1/models defaults to downloaded-only, ?status=all returns everything, /models (no status) still returns everything — 3 passed
  • uv run pytest src/exo/api — 50 passed
  • Confirmed the dashboard's own model browser and chat model selector don't read /v1/models (only dashboard/src/routes/integrations/+page.svelte does, and it already tolerates a shorter list via || []/|| 0 fallbacks)

🤖 Generated with Claude Code

https://claude.ai/code/session_011rjSfwDBTkmySmfU6NgHKF

The OpenAI-compatible /v1/models endpoint was advertising every known
model card, including ones nobody on the cluster has downloaded --
agents/tools that discover models via this endpoint would see and try
models that don't exist locally yet. /models (the dashboard's own
model browser, used for finding new models to download) still needs
to show everything by default, so the two routes needed to diverge;
since they shared one handler whose default couldn't vary per route,
added a thin get_v1_models wrapper that defaults status="downloaded"
and delegates to the existing get_models/status filter. ?status=all
still returns the full list from either route.

Known limitations:
- "Downloaded" is a loose proxy for "usable": a model downloaded to
  disk but not yet placed as a running instance still 404s on first
  use (see API._validate_model_has_instance) -- this only stops
  agents from being offered models that don't exist locally at all,
  it doesn't guarantee the model is ready to serve.
- A multi-node model showing as downloaded only requires completion
  on one node, not all the nodes needed to actually serve it.

Out of scope (separate follow-up, per-model curation checkbox):
this only flips the default filter to downloaded-vs-all. A more
granular "advertise this specific downloaded model to agents" toggle
would need persisted, cluster-synced state (new command + event +
State field + apply() logic) plus dashboard UI, which is a
meaningfully bigger change than this default flip and is left as its
own follow-up.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011rjSfwDBTkmySmfU6NgHKF
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant