Add mandatory inference prefixes based on regions for AWS bedrock models#224
Add mandatory inference prefixes based on regions for AWS bedrock models#224puffitos wants to merge 1 commit intocharmbracelet:mainfrom
Conversation
|
Thanks for this one! Just to confirm, to greatly summarize the changes, this PR essentially adds support for |
|
@puffitos I pushed a few changes on top of yours; can you verify they make sense and that everything's working as expected? I'm also handling the environment variable in Fantasy (our provider and model abstraction layer) to assure that it doesn't fall through the cracks (charmbracelet/fantasy#189). |
|
@meowgorithm thanks for the review and addressing the According to the docs one shouldn't be able to do cross-region inference from an unsupported region (for example when |
|
I'm either not understanding how bedrock works, or the removal of the apac region will break the usage of Claude 4 for some users eventually. The global inference profiles for Claude 4 don't exist in some regions, like Mumbai for example (ap-south-1): ❯ aws bedrock list-inference-profiles --region ap-south-1 \
--query "inferenceProfileSummaries[?contains(inferenceProfileId, 'global')].[inferenceProfileId,inferenceProfileName]" \
--output text | rg "Sonnet 4$"They exist in other regions, like us-east-1: ❯ aws bedrock list-inference-profiles --region us-east-1 \
--query "inferenceProfileSummaries[?contains(inferenceProfileId, 'global')].[inferenceProfileId,inferenceProfileName]" \
--output text | rg "Sonnet 4$"
global.anthropic.claude-sonnet-4-20250514-v1:0 Global Claude Sonnet 4A call to bedrock (just saying hi to Claude), fails as well in ap-south-1 (when using global), but works when using the # us-east-1 has a global inference profile, call works
❯ aws bedrock-runtime converse \
--model-id arn:aws:bedrock:us-east-1:xxxxxxxxxxx:inference-profile/global.anthropic.claude-sonnet-4-20250514-v1:0 \
--messages '[{"role":"user","content":[{"text":"Hey"}]}]' \
--inference-config '{"maxTokens":8192,"stopSequences":[],"temperature":1,"topP":0.999}' \
--additional-model-request-fields '{"top_k":250}' \
--region us-east-1
❯ echo $?
0
# using the fallback global won't work in a region that doesn't have a global profile available
❯ aws bedrock-runtime converse \
--model-id arn:aws:bedrock:ap-south-1:xxxxxxxxxxx:inference-profile/global.anthropic.claude-sonnet-4-20250514-v1:0 \
--messages '[{"role":"user","content":[{"text":"Hey"}]}]' \
--inference-config '{"maxTokens":8192,"stopSequences":[],"temperature":1,"topP":0.999}' \
--additional-model-request-fields '{"top_k":250}' \
--region ap-south-1
aws: [ERROR]: An error occurred (ValidationException) when calling the Converse operation: The provided model identifier is invalid.
Additional error details:
message: The provided model identifier is invalid.
# Using the "correct" apac profile works in the affected region.
❯ aws bedrock-runtime converse \
--model-id arn:aws:bedrock:ap-south-1:xxxxxxxxxxx:inference-profile/apac.anthropic.claude-sonnet-4-20250514-v1:0 \
--messages '[{"role":"user","content":[{"text":"Hey"}]}]' \
--inference-config '{"maxTokens":8192,"stopSequences":[],"temperature":1,"topP":0.999}' \
--additional-model-request-fields '{"top_k":250}' \
--region ap-south-1
❯ echo $?
0I think we need to keep the apac region. |
63c3c8b to
b5abb70
Compare
|
Apologies for dropping off here, @puffitos. I think that makes sense. Do you mind making the change? After that, let's get this and charmbracelet/fantasy#189 merged (if you can take a look at that it would be appreciated too). |
CONTRIBUTING.md.Summary
Resolves #223
AWS_REGION/AWS_DEFAULT_REGION, with per-model region filtering: use the regional prefix if available, fall back toglobal, or exclude the model if neither applies (e.g. Opus 4/4.1 are US-only).TestBedrockConfigRegionsto assert every model inbedrock.jsonhas at least one region configured, and updatedTestBedrockProviderto verify correct model count and prefix per region.Why this approach
Simpler to maintain: one loop, one switch, no extra helpers. The tests validate the config itself, so adding or updating models in
bedrock.jsonwill fail fast if regions are missing.Tradeoff
The
Regionsfield on theModelstruct is currently only used by Bedrock. This feels like feature creep on a shared type, but creating a separate Bedrock-specific config struct seemed like over-engineering for a single field. Happy to hear suggestions from maintainers on a better home for this.Testing done
us-east-1— confirmed that inference profiles are mandatory for all current Anthropic models on Bedrock (direct model ID invocation returnsValidationException). See comment on #223 for details.