Skip to content

Commit 67f7d89

Browse files
committed
feat: make inference provider optional
Inference provider implementations behave inconsistently—some can load normally without an API key (even though it’s required for proper functionality), while others simply fail. To ensure consistency, we should handle inference providers the same way we do vector I/O providers: treat most of them as optional, with vLLM being the exception. Relates to: RHAIENG-1178 Signed-off-by: Sébastien Han <seb@redhat.com>
1 parent f222f3b commit 67f7d89

File tree

1 file changed

+2
-2
lines changed

1 file changed

+2
-2
lines changed

distribution/run.yaml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ providers:
2020
max_tokens: ${env.VLLM_MAX_TOKENS:=4096}
2121
api_token: ${env.VLLM_API_TOKEN:=fake}
2222
tls_verify: ${env.VLLM_TLS_VERIFY:=true}
23-
- provider_id: bedrock-inference
23+
- provider_id: ${env.AWS_ACCESS_KEY_ID:+bedrock}
2424
provider_type: remote::bedrock
2525
config:
2626
aws_access_key_id: ${env.AWS_ACCESS_KEY_ID:=}
@@ -36,7 +36,7 @@ providers:
3636
- provider_id: sentence-transformers
3737
provider_type: inline::sentence-transformers
3838
config: {}
39-
- provider_id: watsonx
39+
- provider_id: ${env.WATSONX_API_KEY:+watsonx}
4040
provider_type: remote::watsonx
4141
config:
4242
url: ${env.WATSONX_BASE_URL:=https://us-south.ml.cloud.ibm.com}

0 commit comments

Comments
 (0)