This image contains the official Open Data Hub Llama Stack distribution, with all the packages and configuration needed to run a Llama Stack server in a containerized environment.
The image is currently shipping with the Open Data Hub version of Llama Stack version 0.3.4+rhai0
You can see an overview of the APIs and Providers the image ships with in the table below.
| API | Provider | External? | Enabled by default? | How to enable |
|---|---|---|---|---|
| agents | inline::meta-reference | No | ✅ | N/A |
| batches | inline::reference | No | ✅ | N/A |
| datasetio | inline::localfs | No | ✅ | N/A |
| datasetio | remote::huggingface | No | ✅ | N/A |
| eval | inline::trustyai_ragas | Yes (version 0.5.1) | ❌ | Set the TRUSTYAI_EMBEDDING_MODEL environment variable |
| eval | remote::trustyai_lmeval | Yes (version 0.4.1) | ✅ | N/A |
| eval | remote::trustyai_ragas | Yes (version 0.5.1) | ❌ | Set the KUBEFLOW_LLAMA_STACK_URL environment variable |
| files | inline::localfs | No | ✅ | N/A |
| files | remote::s3 | No | ❌ | Set the ENABLE_S3 environment variable |
| inference | inline::sentence-transformers | No | ❌ | Set the ENABLE_SENTENCE_TRANSFORMERS environment variable |
| inference | remote::azure | No | ❌ | Set the AZURE_API_KEY environment variable |
| inference | remote::bedrock | No | ❌ | Set the AWS_BEARER_TOKEN_BEDROCK environment variable |
| inference | remote::openai | No | ❌ | Set the OPENAI_API_KEY environment variable |
| inference | remote::vertexai | No | ❌ | Set the VERTEX_AI_PROJECT environment variable |
| inference | remote::vllm | No | ❌ | Set the VLLM_URL environment variable |
| inference | remote::vllm | No | ❌ | Set the VLLM_EMBEDDING_URL environment variable |
| inference | remote::watsonx | No | ❌ | Set the WATSONX_API_KEY environment variable |
| safety | remote::trustyai_fms | Yes (version 0.3.1) | ✅ | N/A |
| scoring | inline::basic | No | ✅ | N/A |
| scoring | inline::braintrust | No | ✅ | N/A |
| scoring | inline::llm-as-judge | No | ✅ | N/A |
| tool_runtime | inline::rag-runtime | No | ✅ | N/A |
| tool_runtime | remote::brave-search | No | ✅ | N/A |
| tool_runtime | remote::model-context-protocol | No | ✅ | N/A |
| tool_runtime | remote::tavily-search | No | ✅ | N/A |
| vector_io | inline::faiss | No | ❌ | Set the ENABLE_FAISS environment variable |
| vector_io | inline::milvus | No | ✅ | N/A |
| vector_io | remote::milvus | No | ❌ | Set the MILVUS_ENDPOINT environment variable |
| vector_io | remote::pgvector | No | ❌ | Set the ENABLE_PGVECTOR environment variable |