Skip to content

feat: allow for more customization around embedding model #157

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mergify merged 1 commit into from
Dec 11, 2025
Merged

feat: allow for more customization around embedding model #157

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion distribution/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ You can see an overview of the APIs and Providers the image ships with in the ta
| batches | inline::reference | No | ✅ | N/A |
| datasetio | inline::localfs | No | ✅ | N/A |
| datasetio | remote::huggingface | No | ✅ | N/A |
| eval | inline::trustyai_ragas | Yes (version 0.5.1) | ❌ | Set the `EMBEDDING_MODEL` environment variable |
| eval | inline::trustyai_ragas | Yes (version 0.5.1) | ❌ | Set the `TRUSTYAI_EMBEDDING_MODEL` environment variable |
| eval | remote::trustyai_lmeval | Yes (version 0.4.1) | ✅ | N/A |
| eval | remote::trustyai_ragas | Yes (version 0.5.1) | ❌ | Set the `KUBEFLOW_LLAMA_STACK_URL` environment variable |
| files | inline::localfs | No | ✅ | N/A |
Expand Down
16 changes: 8 additions & 8 deletions distribution/run.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,18 +122,18 @@ providers:
provider_type: remote::trustyai_lmeval
module: llama_stack_provider_lmeval==0.2.4
config:
use_k8s: ${env.TRUSTYAI_LMEVAL_USE_K8S:=true}
base_url: ${env.VLLM_URL:=http://localhost:8000/v1}
- provider_id: ${env.EMBEDDING_MODEL:+trustyai_ragas_inline}
use_k8s: ${env.TRUSTYAI_LMEVAL_USE_K8S:=true}
base_url: ${env.VLLM_URL:=http://localhost:8000/v1}
- provider_id: ${env.TRUSTYAI_EMBEDDING_MODEL:+trustyai_ragas_inline}
provider_type: inline::trustyai_ragas
module: llama_stack_provider_ragas.inline
config:
embedding_model: ${env.EMBEDDING_MODEL:=}
embedding_model: ${env.TRUSTYAI_EMBEDDING_MODEL:=}
- provider_id: ${env.KUBEFLOW_LLAMA_STACK_URL:+trustyai_ragas_remote}
provider_type: remote::trustyai_ragas
module: llama_stack_provider_ragas.remote
config:
embedding_model: ${env.EMBEDDING_MODEL:=}
embedding_model: ${env.TRUSTYAI_EMBEDDING_MODEL:=}
Copy link
Copy Markdown
Collaborator

@leseb leseb Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how is this model going to be served? i don't see it being optionally registered down below, is it expected?

Copy link
Copy Markdown
Collaborator Author

@nathan-weinberg nathan-weinberg Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was not happening before so AFAIK yes this is expected

kubeflow_config:
results_s3_prefix: ${env.KUBEFLOW_RESULTS_S3_PREFIX:=}
s3_credentials_secret_name: ${env.KUBEFLOW_S3_CREDENTIALS_SECRET_NAME:=}
Expand Down Expand Up @@ -270,10 +270,10 @@ registered_resources:
provider_id: vllm-inference
model_type: llm
- metadata:
embedding_dimension: 768
model_id: granite-embedding-125m
embedding_dimension: ${env.EMBEDDING_DIMENSION:=768}
model_id: ${env.EMBEDDING_MODEL:=granite-embedding-125m-english}
provider_id: ${env.EMBEDDING_PROVIDER:=vllm-embedding}
provider_model_id: ibm-granite/granite-embedding-125m-english
provider_model_id: ${env.EMBEDDING_PROVIDER_MODEL_ID:=ibm-granite/granite-embedding-125m-english}
model_type: embedding
shields: []
vector_dbs: []
Expand Down
Loading