fix(langchain): prefer metadata ls_provider in SummarizationMiddleware token check#36604
Conversation
github-actions
bot
added
fix
org-membership-reviewer
bot
added
trusted-contributor
| "model_provider": "bedrock_converse", | ||
| "ls_provider": "amazon_bedrock", |
There was a problem hiding this comment.
are both these keys in message metadata? I only see model_provider. Do we actually need to update ls_params or message.response_metadata["model_provider"] to make them consistent?
There was a problem hiding this comment.
It's not in a langchain-aws release yet, but ChatBedrock and ChatBedrockConverse will both start adding response_metadata["ls_provider"] = "amazon_bedrock" to outgoing messages with this PR: langchain-ai/langchain-aws#981
model_provider and ls_params can't be directly updated in the Bedrock chat models, as they have distinct values that other mechanisms are dependent on (context here: langchain-ai/langchain-aws#946 (comment)), so this extra metadata field is a workaround to positively identify the base ls_provider directly from messages.
| and ( | ||
| message_provider := last_ai_message.response_metadata.get( | ||
| "ls_provider", | ||
| last_ai_message.response_metadata.get("model_provider"), |
There was a problem hiding this comment.
this is basically, get "ls_provider" and assign to message_provider. if "ls_provider" is not valid then get "model_provider" and assign to message_provider?
| and (message_provider := last_ai_message.response_metadata.get("model_provider")) | ||
| and ( | ||
| message_provider := last_ai_message.response_metadata.get( | ||
| "ls_provider", |
There was a problem hiding this comment.
just curious, for my own learnings what is ls_provider? i understand model/llm provider
3 participants
Fixes langchain-ai/langchain-aws#946
Part 2 of the proposed fix noted in langchain-ai/langchain-aws#946 (comment). The first PR in
langchain-awsaddsresponse_metadata["ls_provider"] = "amazon_bedrock"to messages generated by ChatBedrock and ChatBedrockConverse.This follow-up PR will update
SummarizationMiddleware's_should_summarize_based_on_reported_tokensmethod to preferresponse_metadata["ls_provider"]overmodel_providerwhen checking if a message came from the current provider.This is specifically unblocks optimal use with Bedrock chat models, where Converse API and Invoke API requests require unique
model_providervalues (bedrock/bedrock_converse), but share the same tracingls_provider(amazon_bedrock) for token pricing. For messages from other providers that don't setls_providerin metadata,model_providerwill still be used as the fallback.