Structured Output doesn't work with ChatOpenAI via proxy LiteLLM #34891
Description
Checked other resources
- This is a bug, not a usage question.
- I added a clear and descriptive title that summarizes this issue.
- I used the GitHub search to find a similar question and didn't find it.
- I am sure that this is a bug in LangChain rather than my code.
- The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).
- This is not related to the langchain-community package.
- I posted a self-contained, minimal, reproducible example. A maintainer can copy it and run it AS IS.
Package (Required)
- langchain
- langchain-openai
- langchain-anthropic
- langchain-classic
- langchain-core
- langchain-model-profiles
- langchain-tests
- langchain-text-splitters
- langchain-chroma
- langchain-deepseek
- langchain-exa
- langchain-fireworks
- langchain-groq
- langchain-huggingface
- langchain-mistralai
- langchain-nomic
- langchain-ollama
- langchain-perplexity
- langchain-prompty
- langchain-qdrant
- langchain-xai
- Other / not sure / general
Reproduction Steps / Example Code (Python)
from langchain_openai import ChatOpenAI
from langchain.agents import create_agent
from langchain.output_parsers import ToolStrategy
# Create ChatOpenAI instance pointing to LiteLLM proxy
llm = ChatOpenAI(
api_key="dummy",
base_url="http://<litellm-proxy>",
model="<model-id>",
)
tools = [... any tools]
agent = create_agent(
llm,
tools,
response_format=ToolStrategy(QueryOutput),
)
response = agent.invoke({"messages": [HumanMessage(content="What is the capital of France?")}})Error Message and Stack Trace (if applicable)
- No errors , this is the problem, the output message just looks like this without the structued response
{'messages': [HumanMessage(content='What is the capital of France? Use the Query Response format for your answer {\n "query": "What is the capital of France?",\n "intent": "Find geographic information",\n "action": "search",\n "parameters": {\n "type": "geography"\n }\n}\n', additional_kwargs={}, response_metadata={}, id='f3c70361-24aa-4407-9a8b-f1e2a189c6c7'),
AIMessage(content='\n', additional_kwargs={'refusal': None}, response_metadata={'token_usage': {'completion_tokens': 69, 'prompt_tokens': 624, 'total_tokens': 693, 'completion_tokens_details': None, 'prompt_tokens_details': {'audio_tokens': None, 'cached_tokens': 0, 'cache_creation_tokens': 0}, 'cache_creation_input_tokens': 0, 'cache_read_input_tokens': 0}, 'model_provider': 'openai', 'model_name': 'us.amazon.nova-micro-v1:0', 'system_fingerprint': None, 'id': 'chatcmpl-10fb670f-c8c2-4397-b2bc-aad9202fb5cb', 'finish_reason': 'tool_calls', 'logprobs': None}, id='lc_run--019bfa7f-b175-7190-8c7d-484275c8524d-0', tool_calls=[{'name': 'search_database', 'args': {'query': 'capital of France'}, 'id': 'tooluse_7iUjZlzKRiKDHhbAFSGRhQ', 'type': 'tool_call'}], invalid_tool_calls=[], usage_metadata={'input_tokens': 624, 'output_tokens': 69, 'total_tokens': 693, 'input_token_details': {'cache_read': 0}, 'output_token_details': {}}),
ToolMessage(content='No information found', name='search_database', id='2690e2bd-5926-41f6-8b6e-aa19930ee5e9', tool_call_id='tooluse_7iUjZlzKRiKDHhbAFSGRhQ'),
AIMessage(content="I'm sorry, but I couldn't find the information about the capital of France in the database. However, I can inform you that the capital of France is Paris. If you need any further details or another type of information, feel free to ask!", additional_kwargs={'refusal': None}, response_metadata={'token_usage': {'completion_tokens': 53, 'prompt_tokens': 672, 'total_tokens': 725, 'completion_tokens_details': None, 'prompt_tokens_details': {'audio_tokens': None, 'cached_tokens': 0, 'cache_creation_tokens': 0}, 'cache_creation_input_tokens': 0, 'cache_read_input_tokens': 0}, 'model_provider': 'openai', 'model_name': 'us.amazon.nova-micro-v1:0', 'system_fingerprint': None, 'id': 'chatcmpl-27d0cf51-41e3-42b3-ae11-c6b32f480875', 'finish_reason': 'stop', 'logprobs': None}, id='lc_run--019bfa7f-b632-7b51-b943-d2f626fbdd09-0', tool_calls=[], invalid_tool_calls=[], usage_metadata={'input_tokens': 672, 'output_tokens': 53, 'total_tokens': 725, 'input_token_details': {'cache_read': 0}, 'output_token_details': {}})]}Description
I'm trying to use agents with structured output.
I have a litellm proxy behind bedrock.
When i use open ai directly (without the proxy) the structured output works.
When I use it via proxy it doesn't work and no errors get thrown.
I tried this with amazon nova micro and claude 3 haiku.
System Info
System Information
OS: Darwin
OS Version: Darwin Kernel Version 25.2.0: Tue Nov 18 21:09:45 PST 2025; root:xnu-12377.61.12~1/RELEASE_ARM64_T6030
Python Version: 3.12.7 (main, Dec 10 2024, 07:57:41) [Clang 16.0.0 (clang-1600.0.26.4)]
Package Information
langchain_core: 1.2.7
langchain: 1.2.6
langsmith: 0.6.3
langchain_openai: 1.1.7
langgraph_sdk: 0.3.3
Optional packages not installed
langserve
Other Dependencies
httpx: 0.28.1
jsonpatch: 1.33
langgraph: 1.0.7
openai: 1.109.1
opentelemetry-api: 1.39.1
opentelemetry-exporter-otlp-proto-http: 1.39.1
opentelemetry-sdk: 1.39.1
orjson: 3.11.5
packaging: 24.2
pydantic: 2.12.5
pytest: 8.4.2
pyyaml: 6.0.3
requests: 2.32.5
requests-toolbelt: 1.0.0
rich: 13.9.4
tenacity: 9.1.2
tiktoken: 0.12.0
typing-extensions: 4.15.0
uuid-utils: 0.13.0
zstandard: 0.25.0