Tool calling issue with Amazon Nova models caused by stop sequence behavior

[aristsakpinis93]

Tool calling issue with Amazon Nova models caused by stop sequence behavior

Background

When using the LangChain ReAct agent implementation with Amazon Nova models, there are inconsistencies in handling stop sequences compared to other LLMs (e.g., Claude), affecting tool calling capabilities.

Current Behavior

• LangChain ReAct agent uses ["\nObservation"] as a stop sequence when calling a model
• The aim is to prohibit the model from continued generation. The fields Action and Action Input can then used to call the right tool with the right input (see below in example)
• Nova models include the stop sequence itself in their output, while other models do not. The LangChain implementation is expecting a behavior like the latter.
• This causes issues with the standard ReAct pattern implementation, particularly affecting:
  • Tool calling completion
  • Final answer generation
  • Observation handling

Example

When using the standard ReAct prompt pattern prompt:

Answer the following questions as best you can. You have access to the following tools:

tool(input1: data_type) -> data_type - description
Use the following format:

Question: the input question you must answer
Thought: you should always think about what to do
Action: the action to take, should be one of [...tools...]
Action Input: the input to the action
Observation: the result of the action
... (this Thought/Action/Action Input/Observation can repeat N times)
Thought: I now know the final answer
Final Answer: the final answer to the original input question

Begin!

Question: ...
Thought:

Nova Pro output:
Thought: [thought process]
Action: [action]
Action Input: [input]
Observation:  # <-- Stop sequence included

Claude output (expected by LC):
Thought: [thought process]
Action: [action]
Action Input: [input]
# <-- No stop sequence included

This can be replicated by using the following prompt:

Answer the following questions as best you can. You have access to the following tools:

calculate_total_cost(num_days: int) -> int - Calculates the total cost for the trip. Returns the cost in dollars

Use the following format:

Question: the input question you must answer
Thought: you should always think about what to do
Action: the action to take, should be one of [calculate_total_cost]
Action Input: the input to the action
Observation: the result of the action
... (this Thought/Action/Action Input/Observation can repeat N times)
Thought: I now know the final answer
Final Answer: the final answer to the original input question

Begin!

Question: How much will a 10 day trip cost?
Thought:

[3coins]

@aristsakpinis93
Can you provide a small code sample for us to reproduce this issue.

[aristsakpinis93]

@3coins You can easily reproduce it in the Amazon Bedrock playground, just use the bottom prompt with Nova Pro and 3.7 Sonnet. Is this sufficient? Do you want an actual LC sample?

[3coins]

An actual code sample or link to one would help.

[3coins]

@aristsakpinis93
Here is a sample react agent based on LangGraph's create_react_agent, that seems to work with nova model. I have updated the sample code to match your expected tool. Let us know if this works for your use case.

import random

from langchain_core.tools import tool
from langgraph.prebuilt import create_react_agent

from langchain_aws import ChatBedrockConverse

model = ChatBedrockConverse(
    model="us.amazon.nova-pro-v1:0"
)

@tool
def calculate_total_cost(num_days: int) -> int:
    """Calculates the total cost for the trip. Returns the cost in dollars"""
    return random.randint(1000, 10000)

tools = [calculate_total_cost]
agent = create_react_agent(model, tools=tools)

inputs = {"messages": [("user", "How much will a 10 day trip cost?")]}

for s in agent.stream(inputs, stream_mode="values"):
    message = s["messages"][-1]
    if isinstance(message, tuple):
        print(message)
    else:
        message.pretty_print()

Output

================================ Human Message =================================

How much will a 10 day trip cost?
================================== Ai Message ==================================

[{'type': 'text', 'text': '<thinking> To determine the cost of a 10-day trip, I need to use the `calculate_total_cost` tool. The required argument for this tool is `num_days`, which is 10 in this case. </thinking>\n'}, {'type': 'tool_use', 'name': 'calculate_total_cost', 'input': {'num_days': 10}, 'id': 'tooluse_qAv1U7I_StuUE0D6gfU8pg'}]
Tool Calls:
  calculate_total_cost (tooluse_qAv1U7I_StuUE0D6gfU8pg)
 Call ID: tooluse_qAv1U7I_StuUE0D6gfU8pg
  Args:
    num_days: 10
================================= Tool Message =================================
Name: calculate_total_cost

4125
================================== Ai Message ==================================

The total cost for a 10-day trip is $4125.