Llama Stack 0.0.53 Release

🚀 Initial Release Notes for Llama Stack!

Added

• Resource-oriented design for models, shields, memory banks, datasets and eval tasks
• Persistence for registered objects with distribution
• Ability to persist memory banks created for FAISS
• PostgreSQL KVStore implementation
• Environment variable placeholder support in run.yaml files
• Comprehensive Zero-to-Hero notebooks and quickstart guides
• Support for quantized models in Ollama
• Vision models support for Together, Fireworks, Meta-Reference, and Ollama, and vLLM
• Bedrock distribution with safety shields support
• Evals API with task registration and scoring functions
• MMLU and SimpleQA benchmark scoring functions
• Huggingface dataset provider integration for benchmarks
• Support for custom dataset registration from local paths
• Benchmark evaluation CLI tools with visualization tables
• RAG evaluation scoring functions and metrics
• Local persistence for datasets and eval tasks

Changed

• Split safety into distinct providers (llama-guard, prompt-guard, code-scanner)
• Changed provider naming convention (impls → inline, adapters → remote)
• Updated API signatures for dataset and eval task registration
• Restructured folder organization for providers
• Enhanced Docker build configuration
• Added version prefixing for REST API routes
• Enhanced evaluation task registration workflow
• Improved benchmark evaluation output formatting
• Restructured evals folder organization for better modularity

Removed

• llama stack configure command

What's Changed

• Update download command by @Wauplin in #9
• Update fbgemm version by @jianyuh in #12
• Add CLI reference docs by @dltn in #14
• Added Ollama as an inference impl by @hardikjshah in #20
• Hide older models by @dltn in #23
• Introduce Llama stack distributions by @ashwinb in #22
• Rename inline -> local by @dltn in #24
• Avoid using nearly double the memory needed by @ashwinb in #30
• Updates to prompt for tool calls by @hardikjshah in #29
• RFC-0001-The-Llama-Stack by @raghotham in #8
• Add API keys to AgenticSystemConfig instead of relying on dotenv by @ashwinb in #33
• update cli ref doc by @jeffxtang in #34
• fixed bug in download not enough disk space condition by @sisminnmaw in #35
• Updated cli instructions with additonal details for each subcommands by @varunfb in #36
• Updated URLs and addressed feedback by @varunfb in #37
• Fireworks basic integration by @benjibc in #39
• Together AI basic integration by @Nutlope in #43
• Update LICENSE by @raghotham in #47
• Add patch for SSE event endpoint responses by @dltn in #50
• API Updates: fleshing out RAG APIs, introduce "llama stack" CLI command by @ashwinb in #51
• [inference] Add a TGI adapter by @ashwinb in #52
• upgrade llama_models by @benjibc in #55
• Query generators for RAG query by @hardikjshah in #54
• Add Chroma and PGVector adapters by @ashwinb in #56
• API spec update, client demo with Stainless SDK by @yanxi0830 in #58
• Enable Bing search by @hardikjshah in #59
• add safety to openapi spec by @yanxi0830 in #62
• Add config file based CLI by @yanxi0830 in #60
• Simplified Telemetry API and tying it to logger by @ashwinb in #57
• [Inference] Use huggingface_hub inference client for TGI adapter by @hanouticelina in #53
• Support data: in URL for memory. Add ootb support for pdfs by @hardikjshah in #67
• Remove request wrapper migration by @yanxi0830 in #64
• CLI Update: build -> configure -> run by @yanxi0830 in #69
• API Updates by @ashwinb in #73
• Unwrap ChatCompletionRequest for context_retriever by @yanxi0830 in #75
• CLI - add back build wizard, configure with name instead of build.yaml by @yanxi0830 in #74
• CLI: add build templates support, move imports by @yanxi0830 in #77
• fix prompt with name args by @yanxi0830 in #80
• Fix memory URL parsing by @yanxi0830 in #81
• Allow TGI adaptor to have non-standard llama model names by @hardikjshah in #84
• [API Updates] Model / shield / memory-bank routing + agent persistence + support for private headers by @ashwinb in #92
• Bedrock Guardrails comiting after rebasing the fork by @rsgrewal-aws in #96
• Bedrock Inference Integration by @poegej in #94
• Support for Llama3.2 models and Swift SDK by @ashwinb in #98
• fix safety using inference by @yanxi0830 in #99
• Fixes typo for setup instruction for starting Llama Stack Server section by @abhishekmishragithub in #103
• Make TGI adapter compatible with HF Inference API by @Wauplin in #97
• Fix links & format by @machina-source in #104
• docs: fix typo by @dijonkitchen in #107
• LG safety fix by @kplawiak in #108
• Minor typos, HuggingFace -> Hugging Face by @marklysze in #113
• Reordered pip install and llama model download by @KarthiDreamr in #112
• Update getting_started.ipynb by @delvingdeep in #117
• fix: 404 link to agentic system repository by @moldhouse in #118
• Fix broken links in RFC-0001-llama-stack.md by @bhimrazy in #134
• Validate name in llama stack build by @russellb in #128
• inference: Fix download command in error msg by @russellb in #133
• configure: Fix a error msg typo by @russellb in #131
• docs: Note how to use podman by @russellb in #130
• add env for LLAMA_STACK_CONFIG_DIR by @yanxi0830 in #137
• [bugfix] fix duplicate api endpoints by @yanxi0830 in #139
• Use inference APIs for executing Llama Guard by @ashwinb in #121
• fixing safety inference and safety adapter for new API spec. Pinned t… by @yogishbaliga in #105
• [CLI] remove dependency on CONDA_PREFIX in CLI by @yanxi0830 in #144
• [bugfix] fix #146 by @yanxi0830 in #147
• Extract provider data properly (attempt 2) by @ashwinb in #148
• is_multimodal accepts core_model_id not model itself. by @wizardbc in #153
• fix broken bedrock inference provider by @moritalous in #151
• Fix podman+selinux compatibility by @russellb in #132
• docker: Install in editable mode for dev purposes by @russellb in #160
• [CLI] simplify docker run by @yanxi0830 in #159
• Add a RoutableProvider protocol, support for multiple routing keys by @ashwinb in #163
• docker: Check for selinux before using --security-opt by @russellb in #167
• Adds markdown-link-check and fixes a broken link by @codefromthecrypt in #165
• [bugfix] conda path lookup by @yanxi0830 in #179
• fix prompt guard by @ashwinb in #177
• inference: Add model option to client by @russellb in #170
• [CLI] avoid configure twice by @yanxi0830 in #171
• Check that the model is found before use. by @AshleyT3 in #182
• Add 'url' property to Redis KV config by @Minutis in #192
• Inline vLLM inference provider by @russellb in #181
• add databricks provider by @prithu-dasgupta in #83
• add Weaviate memory adapter by @zainhas in #95
• download: improve help text by @russellb in #204
• Fix ValueError in case chunks are empty by @Minutis in #206
• refactor docs by @yanxi0830 in #209
• README.md: Add vLLM to providers table by @russellb in #207
• Add .idea to .gitignore by @terrytangyuan in #216
• [bugfix] Fix logprobs on meta-reference impl by @yanxi0830 in #213
• Add classifiers in setup.py by @terrytangyuan in #217
• Add function for stopping inference by @kebbbnnn in #224
• JSON serialization for parallel processing queue by @dltn in #232
• Remove "routing_table" and "routing_key" concepts for the user by @ashwinb in #201
• ci: Run pre-commit checks in CI by @russellb in #176
• Fix incorrect completion() signature for Databricks provider by @terrytangyuan in #236
• Enable pre-commit on main branch by @terrytangyuan in #237
• Switch to pre-commit/action by @terrytangyuan in #239
• Remove request arg from chat completion response processing by @terrytangyuan in #240
• Fix broken rendering in Google Colab by @frntn in #247
• Docker compose scripts for remote adapters by @yanxi0830 in #241
• Update getting_started.md by @MeDott29 in #260
• Add llama download support for multiple models with comma-separated list by @tamdogood in #261
• config templates restructure, docs by @yanxi0830 in #262
• [bugfix] fix case for agent when memory bank registered without specifying provider_id by @yanxi0830 in #264
• Add an option to not use elastic agents for meta-reference inference by @ashwinb in #269
• Make all methods async def again; add completion() for meta-reference by @ashwinb in #270
• Add vLLM inference provider for OpenAI compatible vLLM server by @terrytangyuan in #178
• Update event_logger.py by @nehal-a2z in #275
• llama stack distributions / templates / docker refactor by @yanxi0830 in #266
• add more distro templates by @yanxi0830 in #279
• first version of readthedocs by @raghotham in #278
• add completion() for ollama by @dineshyv in #280
• [Evals API] [1/n] Initial API by @yanxi0830 in #287
• Add REST api example for chat_completion by @subramen in #286
• feat: Qdrant Vector index support by @Anush008 in #221
• Add support for Structured Output / Guided decoding by @ashwinb in #281
• [bug] Fix import conflict for SamplingParams by @subramen in #285
• Added implementations for get_agents_session, delete_agents_session and delete_agents by @cheesecake100201 in #267
• [Evals API][2/n] datasets / datasetio meta-reference implementation by @yanxi0830 in #288
• Added tests for persistence by @cheesecake100201 in #274
• Support structured output for Together by @ashwinb in #289
• dont set num_predict for all providers by @dineshyv in #294
• Fix issue w/ routing_table api getting added when router api is not specified by @yanxi0830 in #298
• New quantized models by @ashwinb in #301
• [Evals API][3/n] scoring_functions / scoring meta-reference implementations by @yanxi0830 in #296
• completion() for tgi by @dineshyv in #295
• [enhancement] added templates and enhanced readme by @heyjustinai in #307
• Fix for get_agents_session by @cheesecake100201 in #300
• fix broken --list-templates with adding build.yaml files for packaging by @yanxi0830 in #327
• Added hadamard transform for spinquant by @sacmehta in #326
• [Evals API][4/n] evals with generation meta-reference impl by @yanxi0830 in #303
• completion() for together by @dineshyv in #324
• completion() for fireworks by @dineshyv in #329
• [Evals API][6/n] meta-reference llm as judge, registration for ScoringFnDefs by @yanxi0830 in #330
• update distributions compose/readme by @yanxi0830 in #338
• distro readmes with model serving instructions by @yanxi0830 in #339
• [Evals API][7/n] braintrust scoring provider by @yanxi0830 in #333
• Kill --name from llama stack build by @ashwinb in #340
• Do not cache pip by @stevegrubb in #349
• add dynamic clients for all APIs by @ashwinb in #348
• fix bedrock impl by @dineshyv in #359
• [docs] update documentations by @yanxi0830 in #356
• pgvector fixes by @dineshyv in #369
• persist registered objects with distribution by @dineshyv in #354
• Significantly simpler and malleable test setup by @ashwinb in #360
• Correct a traceback in vllm by @stevegrubb in #366
• add postgres kvstoreimpl by @dineshyv in #374
• add ability to persist memory banks created for faiss by @dineshyv in #375
• fix postgres config validation by @dineshyv in #380
• Enable vision models for (Together, Fireworks, Meta-Reference, Ollama) by @ashwinb in #376
• Kill llama stack configure by @yanxi0830 in #371
• fix routing tables look up key for memory bank by @dineshyv in #383
• add bedrock distribution code by @dineshyv in #358
• Enable remote::vllm by @ashwinb in #384
• Directory rename: providers/impls -> providers/inline, providers/adapters -> providers/remote by @ashwinb in #381
• fix safety signature mismatch by @yanxi0830 in #388
• Remove the safety adapter for Together; we can just use "meta-reference" by @ashwinb in #387
• [LlamaStack][Fireworks] Update client and add unittest by @benjibc in #390
• [bugfix] fix together data validator by @yanxi0830 in #393
• Add provider deprecation support; change directory structure by @ashwinb in #397
• Factor out create_dist_registry by @dltn in #398
• [docs] refactor remote-hosted distro by @yanxi0830 in #402
• [Evals API][10/n] API updates for EvalTaskDef + new test migration by @yanxi0830 in #379
• Resource oriented design for shields by @dineshyv in #399
• Add pip install helper for test and direct scenarios by @dltn in #404
• migrate model to Resource and new registration signature by @dineshyv in #410
• [Docs] Zero-to-Hero notebooks and quick start documentation by @heyjustinai in #368
• Distributions updates (slight updates to ollama, add inline-vllm and remote-vllm) by @ashwinb in #408
• added quickstart w ollama and toolcalling using together by @heyjustinai in #413
• Split safety into (llama-guard, prompt-guard, code-scanner) by @ashwinb in #400
• fix duplicate deploy in compose.yaml by @subramen in #417
• [Evals API][11/n] huggingface dataset provider + mmlu scoring fn by @yanxi0830 in #392
• Folder restructure for evals/datasets/scoring by @yanxi0830 in #419
• migrate memory banks to Resource and new registration by @dineshyv in #411
• migrate dataset to resource by @dineshyv in #420
• migrate evals to resource by @dineshyv in #421
• migrate scoring fns to resource by @dineshyv in #422
• Rename all inline providers with an inline:: prefix by @ashwinb in #423
• fix tests after registration migration & rename meta-reference -> basic / llm_as_judge provider by @yanxi0830 in #424
• fix eval task registration by @yanxi0830 in #426
• fix fireworks data validator by @yanxi0830 in #427
• Allow specifying resources in StackRunConfig by @ashwinb in #425
• Enable sane naming of registered objects with defaults by @ashwinb in #429
• Remove the "ShieldType" concept by @ashwinb in #430
• Inference to use provider resource id to register and validate by @dineshyv in #428
• Kill "remote" providers and fix testing with a remote stack properly by @ashwinb in #435
• add inline:: prefix for localfs provider by @yanxi0830 in #441
• change schema -> dataset_schema for Dataset class by @yanxi0830 in #442
• change schema -> dataset_schema for register_dataset api by @yanxi0830 in #443
• PR-437-Fixed bug to allow system instructions after first turn by @cheesecake100201 in #440
• add support for ${env.FOO_BAR} placeholders in run.yaml files by @ashwinb in #439
• model registration in ollama and vllm check against the available models in the provider by @dineshyv in #446
• Added link to the Colab notebook of the Llama Stack lesson on the Llama 3.2 course on DLAI by @jeffxtang in #445
• make distribution registry thread safe and other fixes by @dineshyv in #449
• local persistent for hf dataset provider by @yanxi0830 in #451
• Support model resource updates and deletes by @dineshyv in #452
• init registry once by @dineshyv in #450
• local persistence for eval tasks by @yanxi0830 in #453
• Fix build configure deprecation message by @hickeyma in #456
• Support parallel downloads for llama model download by @ashwinb in #448
• Add a verify-download command to llama CLI by @ashwinb in #457
• unregister for memory banks and remove update API by @dineshyv in #458
• move hf addapter->remote by @yanxi0830 in #459
• await initialize in faiss by @dineshyv in #463
• fix faiss serialize and serialize of index by @dineshyv in #464
• Extend shorthand support for the llama stack run command by @vladimirivic in #465
• [Agentic Eval] add ability to run agents generation by @yanxi0830 in #469
• Auto-generate distro yamls + docs by @ashwinb in #468
• Allow models to be registered as long as llama model is provided by @dineshyv in #472
• get stack run config based on template name by @dineshyv in #477
• add quantized model ollama support by @wukaixingxp in #471
• Update kotlin client docs by @Riandy in #476
• remove pydantic namespace warnings using model_config by @mattf in #470
• fix llama stack build for together & llama stack build from templates by @yanxi0830 in #479
• Add version to REST API url by @ashwinb in #478
• support adding alias for models without hf repo/sku entry by @dineshyv in #481
• update quick start to have the working instruction by @chuenlok in #467
• add changelog by @dineshyv in #487
• Added optional md5 validate command once download is completed by @varunfb in #486
• Support Tavily as built-in search tool. by @iseeyuan in #485
• Reorganizing Zero to Hero Folder structure by @heyjustinai in #447
• fall to back to read from chroma/pgvector when not in cache by @dineshyv in #489
• register with provider even if present in stack by @dineshyv in #491
• Make run yaml optional so dockers can start with just --env by @ashwinb in #492

New Contributors

• @Wauplin made their first contribution in #9
• @jianyuh made their first contribution in #12
• @dltn made their first contribution in #14
• @hardikjshah made their first contribution in #20
• @raghotham made their first contribution in #8
• @jeffxtang made their first contribution in #34
• @sisminnmaw made their first contribution in #35
• @varunfb made their first contribution in #36
• @benjibc made their first contribution in #39
• @Nutlope made their first contribution in #43
• @hanouticelina made their first contribution in #53
• @rsgrewal-aws made their first contribution in #96
• @poegej made their first contribution in #94
• @abhishekmishragithub made their first contribution in #103
• @machina-source made their first contribution in #104
• @dijonkitchen made their first contribution in #107
• @marklysze made their first contribution in #113
• @KarthiDreamr made their first contribution in #112
• @delvingdeep made their first contribution in #117
• @moldhouse made their first contribution in #118
• @bhimrazy made their first contribution in #134
• @russellb made their first contribution in #128
• @yogishbaliga made their first contribution in #105
• @wizardbc made their first contribution in #153
• @moritalous made their first contribution in #151
• @codefromthecrypt made their first contribution in #165
• @AshleyT3 made their first contribution in #182
• @Minutis made their first contribution in #192
• @prithu-dasgupta made their first contribution in #83
• @zainhas made their first contribution in #95
• @terrytangyuan made their first contribution in #216
• @kebbbnnn made their first contribution in #224
• @frntn made their first contribution in #247
• @MeDott29 made their first contribution in #260
• @tamdogood made their first contribution in #261
• @nehal-a2z made their first contribution in #275
• @dineshyv made their first contribution in #280
• @subramen made their first contribution in #286
• @Anush008 made their first contribution in #221
• @cheesecake100201 made their first contribution in #267
• @heyjustinai made their first contribution in #307
• @sacmehta made their first contribution in #326
• @stevegrubb made their first contribution in #349
• @hickeyma made their first contribution in #456
• @vladimirivic made their first contribution in #465
• @wukaixingxp made their first contribution in #471
• @Riandy made their first contribution in #476
• @mattf made their first contribution in #470
• @chuenlok made their first contribution in #467
• @iseeyuan made their first contribution in #485

Full Changelog: https://github.com/meta-llama/llama-stack/commits/v0.0.53