v0.1.6

0.1.6 Release Notes

Build and Test Agents

• Inference: Fixed support for inline vllm provider
• (New) Agent: Build & Monitor Agent Workflows with Llama Stack + Anthropic's Best Practice Notebook
• (New) Agent: Revamped agent documentation with more details and examples
• Agent: Unify tools and Python SDK Agents API
• Agent: AsyncAgent Python SDK wrapper supporting async client tool calls
• Agent: Support python functions without @client_tool decorator as client tools
• Agent: deprecation for allow_resume_turn flag, and remove need to specify tool_prompt_format
• VectorIO: MilvusDB support added

Agent Evals and Model Customization

• (New) Agent: Llama Stack RAG Lifecycle Notebook
• Eval: Documentation for eval, scoring, adding new benchmarks
• Eval: Distribution template to run benchmarks on llama & non-llama models
• Eval: Ability to register new custom LLM-as-judge scoring functions
• (New) Looking for contributors for open benchmarks. See documentation for details.

Deploy and Monitoring of Agents

• Better support for different log levels across all components for better monitoring

Better Engineering

• Enhance OpenAPI spec to include Error types across all APIs
• Moved all tests to /tests and created unit tests to run on each PR
• Removed all dependencies on llama-models repo

What's Changed

• docs: Add link to distributions guide in quick start guide by @terrytangyuan in #1326
• feat: add prompt-format list by @reidliu41 in #1222
• chore(lint): update Ruff ignores for project conventions and maintainability by @leseb in #1184
• fix: Make remote::vllm compatible with vLLM <= v0.6.3 by @terrytangyuan in #1325
• build(container): misc improvements by @leseb in #1291
• test: cleanup embedding model test suite by @mattf in #1322
• docs: update build doc by @reidliu41 in #1262
• chore: add container cmd check by @reidliu41 in #1306
• chore: update cmd check by @reidliu41 in #1293
• docs: update the downloaded list doc by @reidliu41 in #1266
• fix: replace eval with json decoding by @yanxi0830 in #1327
• feat: enhance OpenAPI spec to include Error types by @ashwinb in #1320
• fix: replace eval with json decoding for format_adapter by @yanxi0830 in #1328
• chore: refactor client tool in test by @yanxi0830 in #1331
• docs: Update llama-stack version in README.md by @suppathak in #1330
• chore: better raise by @ehhuang in #1335
• test: remove old test by @ehhuang in #1334
• docs: update user prompt example by @ehhuang in #1329
• fix: RAG with documents by @ehhuang in #1337
• chore: fix typo by @reidliu41 in #1343
• fix: raise error when request param failed to convert by @ehhuang in #1339
• chore: add container cmd check in start_stack.sh by @reidliu41 in #1340
• fix: don't update tool_config inplace by @ehhuang in #1338
• feat: allow conditionally enabling providers in run.yaml by @ashwinb in #1321
• chore: remove start_venv.sh by @reidliu41 in #1341
• chore: remove dependency on llama_models completely by @ashwinb in #1344
• chore: remove straggler references to llama-models by @ashwinb in #1345
• docs: update Agent documentation by @ehhuang in #1333
• fix: fix pre-commit check issue by @reidliu41 in #1349
• fix: update cmd check logic by @reidliu41 in #1347
• docs: update agent_execution_loop example code by @reidliu41 in #1350
• feat: add a configurable category-based logger by @ashwinb in #1352
• feat: unify max_infer_iters in client/server agent loop by @yanxi0830 in #1309
• fix: Fixed use of chunk.id by @dmartinol in #1356
• refactor: restructure resolver logic and improve type safety by @leseb in #1323
• refactor: move llama3 impl to meta_reference provider by @ashwinb in #1364
• test: introduce recordable mocks for Agent tests by @ehhuang in #1268
• feat: better using get_default_tool_prompt_format by @ehhuang in #1360
• chore: improve --port help text by @reidliu41 in #1346
• refactor: move tests/client-sdk to tests/api by @ashwinb in #1376
• refactor: move more tests, delete some providers tests by @ashwinb in #1382
• feat: deterministic tools ordering by @ehhuang in #1380
• fix: fix end of files for pre-commit by @leseb in #1387
• fix(pgvector): replace hyphens with underscores in table names by @reluctantfuturist in #1385
• docs: fix typo by @reidliu41 in #1390
• refactor(tests): delete inference, safety and agents tests from providers/tests/ by @ashwinb in #1393
• chore: deprecate allow_turn_resume by @yanxi0830 in #1377
• chore: rename task_config to benchmark_config by @yanxi0830 in #1397
• fix: rag as attachment bug by @ehhuang in #1392
• refactor(test): unify vector_io tests and make them configurable by @ashwinb in #1398
• refactor(test): move tools, evals, datasetio, scoring and post training tests by @ashwinb in #1401
• chore: refactor create_and_execute_turn and resume_turn by @yanxi0830 in #1399
• fix: Agent uses the first configured vector_db_id when documents are provided by @dmartinol in #1276
• chore: Update CODEOWNERS by @SLR722 in #1407
• chore: Make README code blocks more easily copy pastable by @ellistarn in #1420
• chore: update hf source for eval notebook by @yanxi0830 in #1403
• docs: api documentation for agents/eval/scoring/datasets by @yanxi0830 in #1400
• feat(1/n): scoring function registration for llm-as-judge by @yanxi0830 in #1405
• docs: fix typo by @reidliu41 in #1416
• chore: Get sqlite_vec and vector_store unit tests passing by @bbrowning in #1413
• fix: regex parser to support more answer formats by @yyymeta in #1425
• feat: effective agent workflow notebook by @yanxi0830 in #1372
• fix: Gracefully handle no choices in remote vLLM response by @bbrowning in #1424
• feat: record token usage for inference API by @dineshyv in #1300
• chore: Reduce flakes in test_text_inference on smaller models by @bbrowning in #1428
• feat: support ClientTool output metadata by @ehhuang in #1426
• fix: don't import from llama_models by @booxter in #1436
• chore: remove unused build dir by @reidliu41 in #1379
• refactor(test): introduce --stack-config and simplify options by @ashwinb in #1404
• ci: add Github workflow which runs unittests in PR by @ashwinb in #1442
• test: revamp eval related integration tests by @yanxi0830 in #1433
• docs: add information on how to set log level before running by @cdoern in #1430
• docs: Add CHANGELOG.md by @terrytangyuan in #1440
• fix: resolve pydantic warning on .dict() usage by @leseb in #1445
• chore: log exception by @ehhuang in #1452
• fix: notebook vision inference by @yanxi0830 in #1423
• docs: improve rag doc by @reidliu41 in #1411
• docs: update eval doc by @yanxi0830 in #1453
• test: recordable mocks use json only by @ehhuang in #1443
• docs: update test_agents to use new Agent SDK API by @ehhuang in #1402
• docs: add back eval concept doc by @yanxi0830 in #1456
• fix: solve ruff B008 warnings by @leseb in #1444
• fix(deps): move chardet and pypdf imports inline where used by @booxter in #1434
• fix: fix agent test recorded responses by @yanxi0830 in #1462
• feat: add Milvus vectorDB by @ashwinb in #1467
• ci: Add script to generate changelog by @terrytangyuan in #1463
• fix: Revert "feat: record token usage for inference API (#1300)" by @dineshyv in #1476
• test: first unit test for resolver by @ashwinb in #1475
• feat: open benchmark template and doc by @SLR722 in #1465
• fix(security): Bump jinja2 to >=3.1.6 by @terrytangyuan in #1461
• chore: Delete unused .gitmodules by @terrytangyuan in #1460
• chore: remove the incorrect output by @reidliu41 in #1472
• chore: add pytest-report.xml to gitignore by @booxter in #1473
• feat: rag eval lifecycle notebook by @yanxi0830 in #1458
• refactor: display defaults in help text by @cdoern in #1480
• feat(agent): plain function as client tool by @ehhuang in #1479
• fix: remove ruff N999 by @leseb in #1388
• feat(logging): implement category-based logging by @leseb in #1362
• build: bump llama-stack-client version by @leseb in #1469
• fix(cli): llama model prompt-format by @ehhuang in #1481
• fix: Swap to AsyncOpenAI client in remote vllm provider by @bbrowning in #1459
• build: include .md by @ehhuang in #1482
• build: add 'tiktoken' to deps by @ehhuang in #1483
• fix: return result for together's get_params by @yanxi0830 in #1484
• ci: enable Dependabot for GitHub Actions by @leseb in #1470
• fix: fix scoring tests by @yanxi0830 in #1487
• chore: escape tool output for logging by @ehhuang in #1490
• feat: updated inline vllm inference provider by @frreiss in #880
• test: image downloading is flaky by @ehhuang in #1491
• test: fix recordable mocks cache key by @ehhuang in #1492
• test: update recorded fixtures by @ehhuang in #1493
• fix: clean up detailed history for CHANGELOG by @yanxi0830 in #1494
• build(deps): bump actions/upload-artifact from 3 to 4 by @dependabot in #1486
• build(deps): bump thollander/actions-comment-pull-request from 2 to 3 by @dependabot in #1485
• fix: Fix open benchmark template by @SLR722 in #1496
• fix: update the open benchmark eval doc by @SLR722 in #1497
• fix: handle log errors by @ehhuang in #1499

New Contributors

• @suppathak made their first contribution in #1330
• @dmartinol made their first contribution in #1356
• @reluctantfuturist made their first contribution in #1385
• @yyymeta made their first contribution in #1425
• @dependabot made their first contribution in #1486

Full Changelog: v0.1.5.1...v0.1.6