feat: Add Bedrock provider support to inference configuration#20
feat: Add Bedrock provider support to inference configuration#20
Conversation
WalkthroughAdded a new inference provider type Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant App as Application
participant Cfg as Config Loader
participant Prov as Provider Registry
participant BR as Bedrock Provider (remote::bedrock)
participant AWS as AWS Bedrock
App->>Cfg: Load providers.inference
Cfg-->>Prov: Register providers (including bedrock-inference)
App->>Prov: getProvider("bedrock-inference")
Prov-->>App: Bedrock Provider instance
App->>BR: infer(request)
BR->>BR: Resolve AWS creds, region, timeouts, retries
BR->>AWS: Invoke model inference (via boto3)
AWS-->>BR: Response / error
BR-->>App: Inference result
note over BR,AWS: Retries governed by retry_mode / total_max_attempts
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Pre-merge checks (3 passed)✅ Passed checks (3 passed)
Poem
Tip 👮 Agentic pre-merge checks are now available in preview!Pro plan users can now enable pre-merge checks in their settings to enforce checklists before merging PRs.
Please see the documentation for more information. Example: reviews:
pre_merge_checks:
custom_checks:
- name: "Undocumented Breaking Changes"
mode: "warning"
instructions: |
Pass/fail criteria: All breaking changes to public APIs, CLI flags, environment variables, configuration keys, database schemas, or HTTP/GraphQL endpoints must be documented in the "Breaking Change" section of the PR description and in CHANGELOG.md. Exclude purely internal or private changes (e.g., code not exported from package entry points or explicitly marked as internal).Please share your feedback with us on this Discord post. 📜 Recent review detailsConfiguration used: CodeRabbit UI Review profile: CHILL Plan: Pro 📒 Files selected for processing (3)
🚧 Files skipped from review as they are similar to previous changes (3)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
✨ Finishing touches🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
skamenan7
requested review from
Elbehery,
VaishnaviHire,
cdoern,
derekhiggins,
leseb,
nathan-weinberg,
rhdedgar and
rhuss
as code owners
|
cc: @leseb , @nathan-weinberg Here is the PR for midstream. Please review. Thanks! |
skamenan7
changed the title
skamenan7
force-pushed
the
feat/Bedrock-midstream-772
branch
from
03d0712 to
5c0df6b
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
distribution/Containerfile (1)
46-49: Do not bake AWS credentials into the image; avoid copying run.yaml with secrets.PR summary indicates run.yaml contains AWS keys. Copying it into the image embeds secrets irreversibly in image layers and caches.
Recommended:
- Remove the COPY of run.yaml and mount it at runtime (Kubernetes Secret/ConfigMap) at /opt/app-root/run.yaml, or
- Keep run.yaml in the image but rely on the AWS default credential chain (IRSA/STS/IMDS/profile) and remove static keys from the file.
Suggested change in this file:
- COPY distribution/run.yaml ${APP_ROOT}/run.yamlNotes:
- Keep the ENTRYPOINT unchanged and mount the config to that exact path in your deployment.
- Also review distribution/providers.d/ contents for secrets; mount those as Secrets if they contain credentials.
🧹 Nitpick comments (1)
distribution/Containerfile (1)
11-11: Pin boto3 and botocore to Bedrock-compatible versionsIn distribution/Containerfile, update the install hunk:
- boto3 \ + boto3==${BOTO3_VERSION} \ + botocore==${BOTOCORE_VERSION} \Add at the top:
ARG BOTO3_VERSION=1.37.0 ARG BOTOCORE_VERSION=1.40.25Ensures support for BedrockRuntime.invoke_model (boto3 v1.37.0) and bedrock-runtime Converse (botocore v1.40.25).
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
distribution/Containerfile(1 hunks)distribution/build.yaml(1 hunks)distribution/run.yaml(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (2)
- distribution/build.yaml
- distribution/run.yaml
|
What is the difference between this and #15? Also, can you please use a fork instead of creating branches directly in the repo? |
|
Sure I will. |
Added remote::bedrock provider to both build.yaml and run.yaml with AWS configuration options including credentials, region, retry settings, and connection timeouts.
leseb
force-pushed
the
feat/Bedrock-midstream-772
branch
from
5c0df6b to
d47ab5f
Compare
I still don't know what the difference is between this and the other PR? |
|
yes, Nathan. This was one the PR where I used a branch instead of fork. |
…madiIram-patch-2 add nudging changes
2 participants
What does this PR do?
Adds AWS Bedrock provider support to the llama-stack distribution by configuring the remote::bedrock provider in both build.yaml and run.yaml files. This enables users to leverage AWS Bedrock models for inference through the llama-stack framework.
The changes include:
remote::bedrockprovider type to build.yaml inference providersSummary by CodeRabbit
New Features
Chores