Skip to content

ci(docs): add external provider info to table #62

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mergify merged 2 commits into from
Oct 7, 2025
Merged

ci(docs): add external provider info to table #62

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
dce45ed
ci(docs): add external provider info to table
nathan-weinberg Oct 7, 2025
05b3df5
ci(docs): use fun emojis so the table is less drab
nathan-weinberg Oct 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions distribution/Containerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,8 @@ RUN pip install \
uvicorn
RUN pip install \
llama_stack_provider_lmeval==0.2.4
RUN pip install \
llama_stack_provider_ragas==0.3.0
RUN pip install \
llama_stack_provider_ragas[remote]==0.3.0
RUN pip install \
Expand Down
54 changes: 27 additions & 27 deletions distribution/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,30 +8,30 @@ The image is currently shipping with upstream Llama Stack version [0.2.22](https

You can see an overview of the APIs and Providers the image ships with in the table below.

| API | Provider | Enabled by default? | How to enable |
|-----|----------|---------------------|---------------|
| agents | inline::meta-reference | Yes | N/A |
| datasetio | inline::localfs | Yes | N/A |
| datasetio | remote::huggingface | Yes | N/A |
| eval | inline::trustyai_ragas | No | Set the `EMBEDDING_MODEL` environment variable |
| eval | remote::trustyai_lmeval | Yes | N/A |
| eval | remote::trustyai_ragas | No | Set the `KUBEFLOW_LLAMA_STACK_URL` environment variable |
| files | inline::localfs | Yes | N/A |
| inference | inline::sentence-transformers | Yes | N/A |
| inference | remote::azure | No | Set the `AZURE_API_KEY` environment variable |
| inference | remote::bedrock | No | Set the `AWS_ACCESS_KEY_ID` environment variable |
| inference | remote::openai | No | Set the `OPENAI_API_KEY` environment variable |
| inference | remote::vertexai | No | Set the `VERTEX_AI_PROJECT` environment variable |
| inference | remote::vllm | No | Set the `VLLM_URL` environment variable |
| inference | remote::watsonx | No | Set the `WATSONX_API_KEY` environment variable |
| safety | remote::trustyai_fms | Yes | N/A |
| scoring | inline::basic | Yes | N/A |
| scoring | inline::braintrust | Yes | N/A |
| scoring | inline::llm-as-judge | Yes | N/A |
| telemetry | inline::meta-reference | Yes | N/A |
| tool_runtime | inline::rag-runtime | Yes | N/A |
| tool_runtime | remote::brave-search | Yes | N/A |
| tool_runtime | remote::model-context-protocol | Yes | N/A |
| tool_runtime | remote::tavily-search | Yes | N/A |
| vector_io | inline::milvus | Yes | N/A |
| vector_io | remote::milvus | No | Set the `MILVUS_ENDPOINT` environment variable |
| API | Provider | External? | Enabled by default? | How to enable |
|-----|----------|-----------|---------------------|---------------|
| agents | inline::meta-reference | No | ✅ | N/A |
| datasetio | inline::localfs | No | ✅ | N/A |
| datasetio | remote::huggingface | No | ✅ | N/A |
| eval | inline::trustyai_ragas | Yes (version 0.3.0) | ❌ | Set the `EMBEDDING_MODEL` environment variable |
| eval | remote::trustyai_lmeval | Yes (version 0.2.4) | ✅ | N/A |
| eval | remote::trustyai_ragas | Yes (version 0.3.0) | ❌ | Set the `KUBEFLOW_LLAMA_STACK_URL` environment variable |
| files | inline::localfs | No | ✅ | N/A |
| inference | inline::sentence-transformers | No | ✅ | N/A |
| inference | remote::azure | No | ❌ | Set the `AZURE_API_KEY` environment variable |
| inference | remote::bedrock | No | ❌ | Set the `AWS_ACCESS_KEY_ID` environment variable |
| inference | remote::openai | No | ❌ | Set the `OPENAI_API_KEY` environment variable |
| inference | remote::vertexai | No | ❌ | Set the `VERTEX_AI_PROJECT` environment variable |
| inference | remote::vllm | No | ❌ | Set the `VLLM_URL` environment variable |
| inference | remote::watsonx | No | ❌ | Set the `WATSONX_API_KEY` environment variable |
| safety | remote::trustyai_fms | Yes (version 0.2.3) | ✅ | N/A |
| scoring | inline::basic | No | ✅ | N/A |
| scoring | inline::braintrust | No | ✅ | N/A |
| scoring | inline::llm-as-judge | No | ✅ | N/A |
| telemetry | inline::meta-reference | No | ✅ | N/A |
| tool_runtime | inline::rag-runtime | No | ✅ | N/A |
| tool_runtime | remote::brave-search | No | ✅ | N/A |
| tool_runtime | remote::model-context-protocol | No | ✅ | N/A |
| tool_runtime | remote::tavily-search | No | ✅ | N/A |
| vector_io | inline::milvus | No | ✅ | N/A |
| vector_io | remote::milvus | No | ❌ | Set the `MILVUS_ENDPOINT` environment variable |
2 changes: 2 additions & 0 deletions distribution/build.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@ distribution_spec:
- provider_type: remote::trustyai_lmeval
module: llama_stack_provider_lmeval==0.2.4
- provider_type: inline::trustyai_ragas
module: llama_stack_provider_ragas==0.3.0
- provider_type: remote::trustyai_ragas
module: llama_stack_provider_ragas[remote]==0.3.0
datasetio:
- provider_type: remote::huggingface
Expand Down
69 changes: 63 additions & 6 deletions scripts/gen_distro_docs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,13 +36,60 @@ def extract_llama_stack_version():
exit(1)


def load_external_providers_info():
"""Load build.yaml and extract external provider information."""
build_yaml_path = REPO_ROOT / "distribution" / "build.yaml"

if not build_yaml_path.exists():
print(f"Error: {build_yaml_path} not found")
exit(1)

try:
with open(build_yaml_path, "r") as file:
build_yaml_data = yaml.safe_load(file)

# Extract providers section from distribution_spec
distribution_spec = build_yaml_data.get("distribution_spec", {})
providers = distribution_spec.get("providers", {})

# Create a mapping of provider_type to external info
external_info = {}

for _, provider_list in providers.items():
if isinstance(provider_list, list):
for provider in provider_list:
if isinstance(provider, dict) and "provider_type" in provider:
provider_type = provider["provider_type"]
module_field = provider.get("module", "")

if module_field:
# Extract version from module field (format: package_name==version)
if "==" in module_field:
# Handle cases like package[extra]==version
version_part = module_field.split("==")[-1]
external_info[provider_type] = (
f"Yes (version {version_part})"
)
else:
external_info[provider_type] = "Yes"

return external_info

except Exception as e:
print(f"Error: Error reading build.yaml: {e}")
exit(1)


def gen_distro_table(providers_data):
# Start with table header
table_lines = [
"| API | Provider | Enabled by default? | How to enable |",
"|-----|----------|---------------------|---------------|",
"| API | Provider | External? | Enabled by default? | How to enable |",
"|-----|----------|-----------|---------------------|---------------|",
]

# Load external provider information from build.yaml
external_providers = load_external_providers_info()

# Create a list to collect all API-Provider pairs for sorting
api_provider_pairs = []

Expand All @@ -61,19 +108,28 @@ def gen_distro_table(providers_data):
)

if conditional_match:
enabled_by_default = "No"
enabled_by_default = "❌"
# Extract the environment variable name (part before :+)
env_var = conditional_match.group(1).split(":+")[0]
# Remove "env." prefix if present
if env_var.startswith("env."):
env_var = env_var[4:]
how_to_enable = f"Set the `{env_var}` environment variable"
else:
enabled_by_default = "Yes"
enabled_by_default = "✅"
how_to_enable = "N/A"

# Determine external status using build.yaml data
external_status = external_providers.get(provider_type, "No")

api_provider_pairs.append(
(api_name, provider_type, enabled_by_default, how_to_enable)
(
api_name,
provider_type,
external_status,
enabled_by_default,
how_to_enable,
)
)

# Sort first by API name, then by provider type
Expand All @@ -83,11 +139,12 @@ def gen_distro_table(providers_data):
for (
api_name,
provider_type,
external_status,
enabled_by_default,
how_to_enable,
) in api_provider_pairs:
table_lines.append(
f"| {api_name} | {provider_type} | {enabled_by_default} | {how_to_enable} |"
f"| {api_name} | {provider_type} | {external_status} | {enabled_by_default} | {how_to_enable} |"
)

return "\n".join(table_lines)
Expand Down
Loading