Default /v1/models to downloaded-only, leave /models unchanged - #2247
Open
alytaphoenix wants to merge 1 commit into
Open
Default /v1/models to downloaded-only, leave /models unchanged#2247alytaphoenix wants to merge 1 commit into
alytaphoenix wants to merge 1 commit into
Conversation
The OpenAI-compatible /v1/models endpoint was advertising every known model card, including ones nobody on the cluster has downloaded -- agents/tools that discover models via this endpoint would see and try models that don't exist locally yet. /models (the dashboard's own model browser, used for finding new models to download) still needs to show everything by default, so the two routes needed to diverge; since they shared one handler whose default couldn't vary per route, added a thin get_v1_models wrapper that defaults status="downloaded" and delegates to the existing get_models/status filter. ?status=all still returns the full list from either route. Known limitations: - "Downloaded" is a loose proxy for "usable": a model downloaded to disk but not yet placed as a running instance still 404s on first use (see API._validate_model_has_instance) -- this only stops agents from being offered models that don't exist locally at all, it doesn't guarantee the model is ready to serve. - A multi-node model showing as downloaded only requires completion on one node, not all the nodes needed to actually serve it. Out of scope (separate follow-up, per-model curation checkbox): this only flips the default filter to downloaded-vs-all. A more granular "advertise this specific downloaded model to agents" toggle would need persisted, cluster-synced state (new command + event + State field + apply() logic) plus dashboard UI, which is a meaningfully bigger change than this default flip and is left as its own follow-up. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011rjSfwDBTkmySmfU6NgHKF
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
/v1/models(the OpenAI-compatible model list, used by external clients/agents/integrations) was advertising every model card exo knows about, including ones nobody on the cluster has downloaded yet. Agents/tools that discover available models via this endpoint would see — and could try to call — models that don't exist locally./models(the dashboard's own model browser, used for finding new models to download) still needs to list everything by default. Both routes shared a singleget_modelshandler, and a query-param default can't diverge per route, so this adds a thinget_v1_modelswrapper registered only on/v1/modelsthat defaultsstatus="downloaded"and delegates to the existing (already-shipped)status=downloadedfilter./modelskeeps its unchanged default.?status=allstill returns the full list from either route — nothing that needs the full list is stranded.Known limitations
API._validate_model_has_instance. This change only stops agents from being offered models that don't exist on the cluster at all; it doesn't guarantee a listed model is actually ready to serve on the first request.Out of scope (separate follow-up)
This is a default-filter flip only. A more granular "advertise this specific downloaded model to agents" checkbox — letting users curate a subset of downloaded models rather than all-or-nothing — would need persisted, cluster-synced state (a new command + event +
Statefield +apply()logic) plus dashboard UI. That's a categorically bigger change than this one and is left as its own follow-up PR; this filter is forward-compatible with it (later: advertised = downloaded AND checked).Test plan
uv run basedpyrighton touched files — 0 errors; full-repo baseline unchanged (319 errors, 110 warnings)uv run ruff check/ruff format— pass/v1/modelsdefaults to downloaded-only,?status=allreturns everything,/models(no status) still returns everything — 3 passeduv run pytest src/exo/api— 50 passed/v1/models(onlydashboard/src/routes/integrations/+page.sveltedoes, and it already tolerates a shorter list via|| []/|| 0fallbacks)🤖 Generated with Claude Code
https://claude.ai/code/session_011rjSfwDBTkmySmfU6NgHKF