Road to v1 - project plan

[leseb]

🚀 Describe the new functionality needed

The following is a high-level plan for the v1 release of the Llama Stack. It is not meant to be a
comprehensive list of all tasks, but rather a guide to help us stay on track. Also, it only covers
what's missing from the current state of the codebase.

Phase 1: Foundation & Infrastructure

CI/CD & Tooling

• Enable integration tests for all APIs (post-training is missing)

Phase 2: API Completion & Stabilization

Missing Features

• Audit and list missing API endpoints
• Identify areas of API we want to change / improve - now is the time to break the API if needed
• Implement missing endpoints + implementations
• Validate consistency with REST standards (like we did for Pagination, this one needs some followup)
• File API initial implementation (feat: add s3 provider to files API #1950???)
• Better MCP support

API Design & Versioning

• Freeze and document contract (schema, fields, defaults) for /v1/ - we have the docs, we just
  need to freeze the contract
• API deprecation strategy planning - more a nice to have?

Error Handling

• Implement consistent error model (exceptions, 422, 404, etc.) - are we doing this already?

Phase 3: Distribution and Provider Integration

Do we need to do anything here?

Phase 4: Testing & Type Safety

Type Safety

• Ensure mypy passes for 100% of codebase

Test Coverage

• Reach 80%+ test coverage for unit tests

Phase 5: Polishing & Release

Code Hygiene

• Idiomatic Python throughout the codebase chore: enable pyupgrade fixes #1806

Documentation

• Project README cleanup: setup, dev, and usage - but more importantly clear and concise

User Experience

• Add make dev / ./scripts/bootstrap.sh to start locally (Implement a 1 line utility to get started with a llama stack distro on local #1374)
• Revisit CLI UX for symmetry and consistency (especially for "llama stack build/run") - a good
  CLI experience is crucial for the adoption of the Llama Stack. (example: llama stack build and llama stack run are not symmetric for --image-name argument when venv is used #1131)
• Remove the ability to llama stack run a container Remove the ability to run the server in container from "llama stack run" #2110
• Rebuild a new UI - make it standalone so any UI launch can connect to any llama-stack server
• Implement XDG-compliant directory handling using the platformdirs library for configuration, data, and cache paths

Server - Security & Observability

• More RBAC capabilities?
• More auth capabilities? (Kubernetes: feat: Add Kubernetes authentication #1778)
• Add dependency vulnerability scanning - we have dependabot already, but is it enough?
• Enable CORS Expose knobs to configure cross-origin resource sharing for server #2119
• Enable multi-tenant capabilities (namespaces)
• Enable multi-users capabilities
• Quotas feat(quota): add server‑side per‑client request quotas (requires auth) #2096
• Repo scanning for vulnerability (with Snyk?)

[bbrowning]

With regards to CI/CD, I'd love for there to be some expectation that providers are regularly tested and known to work. I know that's complicated to arrange, but we don't actually run any CI tests for most providers so it's not always clear when a provider is broken. This doesn't have to be per-PR testing, but for example I run my own nightly runs of lots of inference providers to test and verify OpenAI API compatibility for a lot of them - is there a way for these types of things to get surfaced up to the rest of the community for visibility and fixing if and when specific providers break?

[leseb]

With regards to CI/CD, I'd love for there to be some expectation that providers are regularly tested and known to work. I know that's complicated to arrange, but we don't actually run any CI tests for most providers so it's not always clear when a provider is broken. This doesn't have to be per-PR testing, but for example I run my own nightly runs of lots of inference providers to test and verify OpenAI API compatibility for a lot of them - is there a way for these types of things to get surfaced up to the rest of the community for visibility and fixing if and when specific providers break?

That’s a great point. One approach I’ve used in the past is to assign a clear owner for each provider. That person would be responsible for monitoring the nightly tests and either fixing any issues or, at the very least, reporting that they are broken. The biggest challenge with nightly tests is that they often get neglected without clear ownership. To improve visibility, we could also add badges to the README to show the status of each provider’s nightly tests. We could have a rule that we cannot ship a new release until all the providers pass CI. This will force us to fix the providers in a timely manner.
This also raises an interesting discussion about whether ALL the providers must be CI tested and where the funding might come from if needed.

@bbrowning I’m wondering - is this something we need for v1, or is it more of a general CI improvement? Can we still ship v1 without it?

[bbrowning]

I think the bare minimum we'd want to do around testing providers before shipping a v1 is someone manually running the test suites against each provider to understand what passes and fails and communicating how well that provider works to our community in some way. There is an interesting consideration of whether we hold each provider to the same level of maturity as Llama Stack in general, and if not how we communicate that some providers are better supported than others.

To your larger point about ownership, rules, and funding, I do think the community should start those discussions before v1 so it can communicate to users what's officially supported and known to work vs what hopefully works vs what is entirely supported by a 3rd party and no official statement of working or not comes from Llama Stack. Some of this may be in-tree vs external providers, and some of this may be what's tested and gating before merging PRs vs what's run nightly or less often with only best-effort monitoring and maintenance.

[leseb]

@bbrowning reflected that discussion in the summary, thanks!

[franciscojavierarceo]

@leseb we discussed a key point about this during the community call today and an action item was for us to define a concise vision for what we want for v1. And it should be grounded in what we enable in the project (more notes here).

@ashwinb suggested we put It front and center as the goal.

I like the framing as I think it'll help us rally towards a common goal.

[booxter]

Consider these for the v1 list.

• For the platform to be cohesive, we need a way to pass over artifacts between flows. Basic stories like: "A tuned model should be uploaded to storage. Then picked up from there for inference."; "Synthetic data was generated. It was uploaded to storage. It was then picked up for fine tuning." For this, /files/ API should become an actual thing (right now all we have are API stubs).
• Bless v1 APIs in docs. Enable integration tests for each supported API flow. (I'm specifically thinking of post-training that currently has no way to get access to llama models in CI - that I know of - and hence is not validated. There may be more.)
• On the other hand, temporarily withdraw APIs that we are not comfortable with. (e.g. because we think they are immature or there are no acceptable providers implemented for them.) Withdrawal may mean removal or marking them out of v1. This ties to API deprecation story: additionally, there should be a clear path for API evolution (like alpha/beta/staging phases for graduation.)
• Community should become more self-sufficient. I'm not talking about "Foundation", but some basic governance structure / rules of engagement, bug triage and similar routines; a more diverse core team; control over SDK generation (Stainless); getting to a point where we are satisfied with the final location of repos (e.g. do we move SDK and other repos to the new llamastack github organization before v1 to avoid later churn).
• (Less sure about this one) Improve test coverage for MacOS which is a popular platform among developers. We can start with running unit tests and then enable some integration flows for basic flows like inference with ollama and such.

[bbrowning]

As we talk about a v1 and inference APIs, we should probably decide whether we want to support both the Llama Stack specific Inference API and our OpenAI APIs in perpetuity or phase one out. And, if so, make sure we're happy with the url for each of those API endpoints (ie the duplicated "v1" in the OpenAI ones).

Likewise, if want to support OpenAI-compatible inference APIs for v1, does that include Completions, Chat Completions, and Responses? Or just a subset of those? I'd like us to support all three of those OpenAI-compatible APIs so that Llama Stack has maximum compatibility with existing applications, but know that Responses is still in its early form so we'd need to push to get that stabilized quickly.

[derekhiggins]

For remote providers would there be appetite for implementing cached response testing for provider verification tests?

I've been experimenting with this with a prototype e.g. for OpenAI tests:

(venv) (base) derekh@laptop:~/workarea/llama-stack$ pytest tests/verifications/openai_api/ --provider=openai
============================================================================================================ test session starts ============================================================================================================
platform linux -- Python 3.13.2, pytest-8.3.5, pluggy-1.5.0
rootdir: /home/derekh/workarea/llama-stack
configfile: pyproject.toml
plugins: json-report-1.5.0, anyio-4.9.0, metadata-3.1.1
collected 92 items

tests/verifications/openai_api/test_chat_completion.py ............................................................................ [ 82%]
tests/verifications/openai_api/test_responses.py ................ [100%]

============================================================================================================ 92 passed in 2.74s =============================================================================================================

After the initial run, subsequent test executions complete in ~3s versus ~2m45s for live API calls.
If the test runners share persistent storage, we could implement this on all PR's with minimal changes without blowing budgets.
The cache could have configurable TTL settings to periodically refresh against live endpoints.

Much faster test execution, Reduced API costs, More consistent test environments
But the potential to miss recent provider-side changes/breaks

Happy to elaborate if there's interest

[derekhiggins]

For remote providers would there be appetite for implementing cached response testing for provider verification tests?

I've been experimenting with this with a prototype e.g. for OpenAI tests:

(venv) (base) derekh@laptop:~/workarea/llama-stack$ pytest tests/verifications/openai_api/ --provider=openai
============================================================================================================ test session starts ============================================================================================================
platform linux -- Python 3.13.2, pytest-8.3.5, pluggy-1.5.0
rootdir: /home/derekh/workarea/llama-stack
configfile: pyproject.toml
plugins: json-report-1.5.0, anyio-4.9.0, metadata-3.1.1
collected 92 items
tests/verifications/openai_api/test_chat_completion.py ............................................................................ [ 82%]
tests/verifications/openai_api/test_responses.py ................ [100%]
============================================================================================================ 92 passed in 2.74s =============================================================================================================

After the initial run, subsequent test executions complete in ~3s versus ~2m45s for live API calls. If the test runners share persistent storage, we could implement this on all PR's with minimal changes without blowing budgets. The cache could have configurable TTL settings to periodically refresh against live endpoints.

Much faster test execution, Reduced API costs, More consistent test environments But the potential to miss recent provider-side changes/breaks

Happy to elaborate if there's interest

I've put together an example of what I was referring to (on a fork so that I could set a API key as a secret)
This PR derekhiggins#1 runs the openai verification tests,
this run (without a cache) typically takes 3 to 5 minutes and frequently fails (due to server side errors or model response inconsistencies)
https://github.com/derekhiggins/llama-stack/actions/runs/14974894740/job/42064600388

...
FAILED tests/verifications/openai_api/test_responses.py::test_response_non_streaming_multi_turn_image[gpt-4o-llama_image_search] - assert 'model' in 'the search for "llama maverick scout" yielded several results, primarily related to products named "maverick scout," including knives and rc vehicles. however, there is no direct connection between these products and llamas. additionally, there is no specific information linking the terms "maverick" and "scout" to any particular llama breed or individual. '
 +  where 'model' = <built-in method lower of str object at 0x7fbb4c2387f0>()
 +    where <built-in method lower of str object at 0x7fbb4c2387f0> = 'model'.lower
=================== 1 failed, 91 passed in 180.69s (0:03:00) ===================

But when the cache is used, the tests take around 3 seconds, cost nothing (after the initial success) and aren't
subject to server side API problems or model inconsistencies

https://github.com/derekhiggins/llama-stack/actions/runs/14975034016/job/42065066576?pr=1

...
tests/verifications/openai_api/test_responses.py::test_response_non_streaming_image[gpt-4o-mini-llama_image] PASSED [ 97%]
tests/verifications/openai_api/test_responses.py::test_response_non_streaming_multi_turn_image[gpt-4o-llama_image_search] PASSED [ 98%]
tests/verifications/openai_api/test_responses.py::test_response_non_streaming_multi_turn_image[gpt-4o-mini-llama_image_search] PASSED [100%]

============================== 92 passed in 3.67s ==============================