Skip to content

chore: resync downstream content #25

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
leseb merged 8 commits into from
Sep 12, 2025
Merged

chore: resync downstream content #25

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
358fe45
chore: add files API to the definition
leseb Sep 12, 2025
b770d1a
feat: add milvus remote provider
leseb Sep 12, 2025
d576cb0
chore: add milvus remote to the build config
leseb Sep 12, 2025
7e76164
feat: add files localfs provider
leseb Sep 12, 2025
cbbc46c
feat: add postgres sql dependencies
leseb Sep 12, 2025
6c5211b
chore: bump lmeval version
leseb Sep 12, 2025
7674ef8
fix: allow trustyai lmeval to be disabled
leseb Sep 12, 2025
c16eebb
chore: convert build.yaml to use modules for external providers
leseb Sep 12, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions distribution/Containerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,14 @@ WORKDIR /opt/app-root
RUN pip install sqlalchemy # somehow sqlalchemy[asyncio] is not sufficient
RUN pip install \
aiosqlite \
asyncpg \
autoevals \
boto3 \
chardet \
datasets>=4.0.0 \
fastapi \
fire \
httpx \
kubernetes \
llama_stack_provider_lmeval==0.2.4 \
llama_stack_provider_trustyai_fms==0.2.1 \
matplotlib \
mcp>=1.8.1 \
Expand All @@ -39,6 +38,8 @@ RUN pip install \
tqdm \
transformers \
uvicorn
RUN pip install \
llama_stack_provider_lmeval==0.2.3
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot Sep 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Create directories used by the new Files API and DB paths

run.yaml writes under /opt/app-root/src/.llama/distributions/rh (and files/). Pre-create to avoid runtime failures on first boot.

Apply after the existing mkdir step:

 RUN pip install --no-cache llama-stack==0.2.21
-RUN mkdir -p ${HOME}/.llama/providers.d ${HOME}/.cache
+RUN mkdir -p ${HOME}/.llama/providers.d ${HOME}/.cache \
+    && mkdir -p /opt/app-root/src/.llama/distributions/rh/files

Committable suggestion skipped: line range outside the PR's diff.

🤖 Prompt for AI Agents 
In distribution/Containerfile around lines 41 to 42, the Dockerfile installs
llama_stack_provider_lmeval but does not pre-create directories used by the new
Files API and DB paths; add a step (immediately after the existing mkdir step)
to create /opt/app-root/src/.llama/distributions/rh and
/opt/app-root/src/.llama/files (or the exact files/ path used by run.yaml), set
appropriate ownership (app-root user) and permissions so the runtime process can
write there, and ensure the mkdir command uses -p to avoid errors.

RUN pip install --extra-index-url https://download.pytorch.org/whl/cpu torch torchao>=0.12.0 torchvision
RUN pip install --no-deps sentence-transformers
RUN pip install --no-cache llama-stack==0.2.21
Expand Down
6 changes: 6 additions & 0 deletions distribution/build.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,14 @@ distribution_spec:
- provider_type: inline::sentence-transformers
vector_io:
- provider_type: inline::milvus
- provider_type: remote::milvus
safety:
- provider_type: remote::trustyai_fms
agents:
- provider_type: inline::meta-reference
eval:
- provider_type: remote::trustyai_lmeval
module: llama_stack_provider_lmeval==0.2.3
datasetio:
- provider_type: remote::huggingface
- provider_type: inline::localfs
Expand All @@ -28,10 +30,14 @@ distribution_spec:
- provider_type: remote::tavily-search
- provider_type: inline::rag-runtime
- provider_type: remote::model-context-protocol
files:
- provider_type: inline::localfs
container_image: registry.redhat.io/ubi9/python-311:9.6-1749631027
additional_pip_packages:
- aiosqlite
- sqlalchemy[asyncio]
- asyncpg
- psycopg2-binary
image_type: container
image_name: llama-stack-rh
external_providers_dir: distribution/providers.d
17 changes: 16 additions & 1 deletion distribution/run.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ apis:
- telemetry
- tool_runtime
- vector_io
- files
providers:
inference:
- provider_id: vllm-inference
Expand Down Expand Up @@ -44,6 +45,20 @@ providers:
type: sqlite
namespace: null
db_path: /opt/app-root/src/.llama/distributions/rh/milvus_registry.db
- provider_id: ${env.MILVUS_ENDPOINT:+milvus-remote}
provider_type: remote::milvus
config:
uri: ${env.MILVUS_ENDPOINT:=}
token: ${env.MILVUS_TOKEN:=}
secure: ${env.MILVUS_SECURE:=}
consistency_level: ${env.MILVUS_CONSISTENCY_LEVEL:=}
ca_pem_path: ${env.MILVUS_CA_PEM_PATH:=}
client_pem_path: ${env.MILVUS_CLIENT_PEM_PATH:=}
client_key_path: ${env.MILVUS_CLIENT_KEY_PATH:=}
kvstore:
type: sqlite
namespace: null
db_path: /opt/app-root/src/.llama/distributions/rh/milvus_remote_registry.db
safety:
- provider_id: trustyai_fms
provider_type: remote::trustyai_fms
Expand All @@ -66,7 +81,7 @@ providers:
- provider_id: trustyai_lmeval
provider_type: remote::trustyai_lmeval
config:
use_k8s: True
use_k8s: ${env.TRUSTYAI_LMEVAL_USE_K8S:=true}
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot Sep 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Verification agent

🧩 Analysis chain

Critical: env var name and default for use_k8s

PR text references LM_EVAL_USE_K8S=false, but config reads TRUSTYAI_LMEVAL_USE_K8S with default true. With kubernetes lib not installed in the image, default true can cause runtime errors.

Apply one of the following (preferred: align with PR doc and default to false):

-        use_k8s: ${env.TRUSTYAI_LMEVAL_USE_K8S:=true}
+        use_k8s: ${env.LM_EVAL_USE_K8S:=false}

Run to confirm naming and defaults across the repo:

🏁 Script executed:

#!/bin/bash
set -euo pipefail
echo "Containerfile base:"
rg -n '^FROM .*python-(3\d+)' -n distribution/Containerfile
echo
echo "build.yaml container_image:"
rg -n 'container_image:' -n distribution/build.yaml
echo
echo "Env var occurrences:"
rg -n 'LM_EVAL_USE_K8S|TRUSTYAI_LMEVAL_USE_K8S' -n

Length of output: 403

Critical: env var name and default for use_k8s

distribution/run.yaml defaults TRUSTYAI_LMEVAL_USE_K8S to true (distribution/run.yaml:84); tests/smoke.sh sets TRUSTYAI_LMEVAL_USE_K8S=False (tests/smoke.sh:14). LM_EVAL_USE_K8S is not present. Default=true can cause runtime errors when k8s libs are not in the image. Apply one of the following (preferred: align with PR doc and default to false):

-        use_k8s: ${env.TRUSTYAI_LMEVAL_USE_K8S:=true}
+        use_k8s: ${env.LM_EVAL_USE_K8S:=false}

Files: distribution/run.yaml:84, tests/smoke.sh:14

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
use_k8s: ${env.TRUSTYAI_LMEVAL_USE_K8S:=true}
use_k8s: ${env.LM_EVAL_USE_K8S:=false}
🤖 Prompt for AI Agents 
In distribution/run.yaml around line 84, the environment variable and default
are wrong: change the key to use LM_EVAL_USE_K8S (matching the PR doc) and set
the default to false instead of true; ensure tests/smoke.sh (line ~14) and any
other refs use the same variable name and casing (or update the test to export
LM_EVAL_USE_K8S=False) so runtime defaults do not enable k8s by default.

base_url: ${env.VLLM_URL:=http://localhost:8000/v1}
datasetio:
- provider_id: huggingface
Expand Down
1 change: 0 additions & 1 deletion tests/smoke.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@ function start_and_wait_for_llama_stack_container {
--env INFERENCE_MODEL="$INFERENCE_MODEL" \
--env VLLM_URL="$VLLM_URL" \
--env TRUSTYAI_LMEVAL_USE_K8S=False \
--env TRUSTYAI_LM_EVAL_NAMESPACE=dummy \
--name llama-stack \
"$IMAGE_NAME:$GITHUB_SHA"
echo "Started Llama Stack container..."
Expand Down
Loading