Help understanding bedrock, reasoning and inference profiles

[icsy7867]

I am trying to understand the different options. I am using bedrock/govcloud and using custom inference profiles.

I originally had something like this. But what I found were tools like Claude Code and Claude Desktop had trouble identifying the model, and in this case tried to apply Adaptive reasoning which would fail for Sonnet 4.5

      - model_name: model-a
        litellm_params:
          model: bedrock/converse/arn:aws-us-gov:bedrock:us-gov-east-1:537798193888:application-inference-profile/xxxxxxxxxx
          ...

Reading through the documentation, I am trying to apply the custom inference profile and the model ARN using the model_id and model as the documentation showed. However now it appears that this isnt using the inference profile at all? I am still digging into this one.

      - model_name: model-b
        litellm_params:
          model: bedrock/converse/arn:aws-us-gov:bedrock:us-gov-east-1::foundation-model/anthropic.claude-sonnet-4-5-20250929-v1:0
          model_id: arn:aws-us-gov:bedrock:us-gov-east-1:537798193888:application-inference-profile/xxxxxxxxxxxxxxxxx
          ...

I also had someone report back to be some interesting results using Pi code.

With the openai-completions API:
Claude Sonnet 4.5 works with reasoning: true . Can alter the thinking level, and I can see the thinking output (only has the model set)
Claude Sonnet 5 only works with reasoning: false. (This has the model and model_id set)

With the anthropic-messages API:
Claude Sonnet 4.5 only works with reasoning: false
Claude Sonnet 5 works with reasoning: true, But not reasoning output. (I think this makes since with the bedrock/anthropic style of reasoning.)

Does anyone know what I SHOULD be doing? :D

Thank you!

[Ganesh-403]

@icsy7867
I think there are two separate concerns getting mixed together here:

1. How Bedrock routes the request (foundation model vs. application inference profile)
2. How clients infer model capabilities (reasoning, thinking blocks, etc.)

For the first point, the current LiteLLM documentation recommends using the foundation model ARN in model and the Application Inference Profile ARN in model_id, e.g.:

litellm_params:
  model: bedrock/converse/arn:aws-us-gov:bedrock:...:foundation-model/anthropic.claude-sonnet-4-5-...
  model_id: arn:aws-us-gov:bedrock:...:application-inference-profile/...

This matches how AWS describes using application inference profiles: the inference profile ARN is supplied as the modelId when invoking the Bedrock Converse APIs, while the underlying foundation model remains identifiable. :contentReference[oaicite:0]{index=0}

The advantage of this approach is that clients (Claude Code, Claude Desktop, Pi Code, etc.) can still identify the underlying model family (claude-sonnet-4.5) and make capability decisions accordingly, rather than seeing an opaque inference-profile ARN.

Regarding the reasoning differences:

• OpenAI-compatible API and Anthropic Messages API don't expose reasoning in exactly the same way.
• On Bedrock, Anthropic's "thinking" feature is also translated through Bedrock's Converse API, so it's expected that the observable behavior won't be identical across both protocols. I couldn't find documentation guaranteeing feature parity between the two interfaces. :contentReference[oaicite:1]{index=1}

One thing I'd verify is whether the request is actually using your application inference profile. You can usually confirm this from the AWS side (CloudWatch metrics or Cost Explorer tags), since Application Inference Profiles are specifically designed for usage attribution. :contentReference[oaicite:2]{index=2}

One question for the maintainers:

When both model (foundation model ARN) and model_id (Application Inference Profile ARN) are specified, does LiteLLM always invoke Bedrock using the inference profile ARN under the hood, or are there request paths where model_id is ignored?

The docs suggest it should be used, but confirming that behavior would help explain whether the issue is with inference profile routing or with how downstream clients detect reasoning capabilities.

If this solves your problem, feel free to mark it as the accepted answer so others can find it easily.

[icsy7867]

Thank you for the reply! It does seem to get mixed together for me. I was using the full ARN for model as I found when using cloud shell.

model: bedrock/converse/arn:aws-us-gov:bedrock:us-gov-east-1::foundation-model/anthropic.claude-sonnet-4-5-20250929-v1:0

However the cost explorer/dashboards were showing almost all of my traffic using this not being correctly tagged.

I tried to assign this this way yesterday:

      - model_name: claude-opus-4.8
        litellm_params:
          model: bedrock/converse/anthropic.claude-opus-4-8
          model_id: arn:aws-us-gov:bedrock:us-gov-east-1:537798193888:application-inference-profile/fiz59bl0ikyt
          ...

And now ALL my request are getting correctly tagged through the custom inference profile.

[unusdon]

Two things are getting tangled here — the ARN split for inference profiles is fine as you have it, but the reasoning: true vs thinking: {...} asymmetry is what's actually causing the cross-endpoint behavior difference.

1. ARN split — your model-b config is right

• model: bedrock/converse/arn:...:foundation-model/anthropic.claude-sonnet-4-5-... — foundation-model ARN. LiteLLM uses this to identify the model family, so clients (Claude Code, Claude Desktop, Pi Code) can make capability decisions.
• model_id: arn:...:application-inference-profile/... — inference-profile ARN. LiteLLM passes this as the modelId on the Bedrock Converse call, so cost tracking / tag routing hits your profile.

To confirm the profile is actually being invoked, tag it in AWS and watch CloudWatch → AWS/Bedrock namespace, filter by the bedrock-application-inference-profile-arn dimension. If invocations show up under your ARN there, routing is correct.

2. Why reasoning: true behaves differently on each endpoint

This is unrelated to the inference profile — it's about which LiteLLM endpoint the client hits:

Endpoint	Parameter LiteLLM expects	What happens
/v1/chat/completions (openai-completions)	reasoning: true or reasoning_effort	Translated into the target provider's native form. For Anthropic-on-Bedrock, LiteLLM injects a thinking: {type: enabled, budget_tokens: N} block with a default budget.
/v1/messages (anthropic-messages)	thinking: {type: enabled, budget_tokens: N}	Native Anthropic pass-through. reasoning: true isn't the Anthropic-family reasoning key — the messages API doesn't recognize it, so it either no-ops or the model backend rejects it, depending on version.

So: Sonnet 4.5 "works with reasoning: true" on openai-completions because LiteLLM did the translation; the same request on anthropic-messages fails because you were passing an OpenAI-family key on an Anthropic-family endpoint.

3. Recommended config

If your clients are a mix of OpenAI-style and Anthropic-style consumers, publish two model_name entries for the same underlying inference profile — one per convention:

model_list:
  # For openai-completions clients (Claude Code, Pi Code, anything using /v1/chat/completions)
  - model_name: sonnet-4-5-oai
    litellm_params:
      model: bedrock/converse/arn:aws-us-gov:bedrock:us-gov-east-1::foundation-model/anthropic.claude-sonnet-4-5-20250929-v1:0
      model_id: arn:aws-us-gov:bedrock:us-gov-east-1:537798193888:application-inference-profile/xxxxxxxx
      # clients pass reasoning: true / reasoning_effort per-request

  # For anthropic-messages clients (direct SDK users, anything using /v1/messages)
  - model_name: sonnet-4-5-anthropic
    litellm_params:
      model: bedrock/converse/arn:aws-us-gov:bedrock:us-gov-east-1::foundation-model/anthropic.claude-sonnet-4-5-20250929-v1:0
      model_id: arn:aws-us-gov:bedrock:us-gov-east-1:537798193888:application-inference-profile/xxxxxxxx
      thinking:
        type: enabled
        budget_tokens: 8000

Point each client at the alias matching its native convention. Adding drop_params: true at the global level helps too — it silently drops reasoning: true if it leaks onto the anthropic-messages path instead of raising.

4. On the "reasoning output visible on one path but not the other"

The openai-completions path serialises Anthropic's thinking block into a reasoning_content field on the streamed chunk (that's why you see it). The anthropic-messages path preserves the native thinking content block, which clients written against the raw Anthropic Messages schema render, but clients that only render content[].text blocks silently drop.