ProviderStrategy fails with JSONDecodeError for Gemini on VertexAI

[alejandroem1]

Checked other resources

• This is a bug, not a usage question.
• I added a clear and descriptive title that summarizes this issue.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).
• This is not related to the langchain-community package.
• I posted a self-contained, minimal, reproducible example. A maintainer can copy it and run it AS IS.

Package (Required)

• langchain
• langchain-openai
• langchain-anthropic
• langchain-classic
• langchain-core
• langchain-cli
• langchain-model-profiles
• langchain-tests
• langchain-text-splitters
• langchain-chroma
• langchain-deepseek
• langchain-exa
• langchain-fireworks
• langchain-groq
• langchain-huggingface
• langchain-mistralai
• langchain-nomic
• langchain-ollama
• langchain-perplexity
• langchain-prompty
• langchain-qdrant
• langchain-xai
• Other / not sure / general

Example Code (Python)

"""
Minimal reproducible example showing ProviderStrategy JSONDecodeError with Gemini.

This script demonstrates:
1. ToolStrategy works perfectly ✅
2. ProviderStrategy fails with JSONDecodeError ❌
3. Both use the same model, schema, and prompt

Run with:
    export GCP_PROJECT_ID="your-project-id"
    python reproduce_provider_strategy_bug.py

Requirements:
    pip install langchain langchain-google-vertexai
"""
import asyncio
import os
from langchain_google_vertexai import ChatVertexAI
from langchain.agents import create_agent
from langchain.agents.structured_output import ProviderStrategy, ToolStrategy

# Simple JSON schema for testing
SCHEMA = {
    "type": "object",
    "properties": {
        "sentiment": {
            "type": "string",
            "enum": ["positive", "negative", "neutral"],
            "description": "The sentiment of the text"
        },
        "confidence": {
            "type": "number",
            "minimum": 0,
            "maximum": 1,
            "description": "Confidence score"
        }
    },
    "required": ["sentiment", "confidence"],
    "additionalProperties": False
}

PROMPT = "Analyze this review: 'This product is absolutely amazing!'"


async def test_tool_strategy_works():
    """✅ This works - proves setup is correct"""
    print("\n" + "=" * 80)
    print("TEST 1: ToolStrategy with Gemini")
    print("=" * 80)

    model = ChatVertexAI(
        model_name="gemini-2.5-flash",
        project=os.getenv("GCP_PROJECT_ID"),
        location="us-central1",
        temperature=0.0,
    )

    agent = create_agent(
        model=model,
        tools=[],
        response_format=ToolStrategy(schema=SCHEMA, handle_errors=True),
    )

    try:
        result = await agent.ainvoke({
            "messages": [{"role": "user", "content": PROMPT}]
        })

        print("✅ SUCCESS - ToolStrategy works!")
        print(f"Structured response: {result.get('structured_response')}")
        print(f"Response type: {type(result.get('structured_response'))}")
    except Exception as e:
        print(f"❌ UNEXPECTED FAILURE: {type(e).__name__}")
        print(f"Error: {str(e)}")


async def test_provider_strategy_fails():
    """❌ This fails - JSONDecodeError at char 0"""
    print("\n" + "=" * 80)
    print("TEST 2: ProviderStrategy with Gemini")
    print("=" * 80)

    model = ChatVertexAI(
        model_name="gemini-2.5-flash",
        project=os.getenv("GCP_PROJECT_ID"),
        location="us-central1",
        temperature=0.0,
    )

    try:
        agent = create_agent(
            model=model,
            tools=[],
            response_format=ProviderStrategy(schema=SCHEMA),  # ❌ This fails
        )

        result = await agent.ainvoke({
            "messages": [{"role": "user", "content": PROMPT}]
        })

        print("✅ UNEXPECTED SUCCESS!")
        print(f"Structured response: {result.get('structured_response')}")
        print("(If you see this, the bug may have been fixed!)")

    except Exception as e:
        print(f"❌ FAILED: {type(e).__name__}")
        print(f"Error message: {str(e)}")
        print()
        print("Full traceback:")
        import traceback
        traceback.print_exc()


async def main():
    """Run both tests to demonstrate the issue"""
    print("\n" + "=" * 80)
    print("BUG REPRODUCTION: ProviderStrategy JSONDecodeError with Gemini")
    print("=" * 80)

    # Check environment
    if not os.getenv("GCP_PROJECT_ID"):
        print()
        print("ERROR: GCP_PROJECT_ID environment variable not set")
        print()
        print("Please set it before running:")
        print("  export GCP_PROJECT_ID='your-project-id'")
        print()
        return

    print(f"Project ID: {os.getenv('GCP_PROJECT_ID')}")
    print(f"Model: gemini-2.5-flash")
    print(f"Schema: Simple sentiment analysis")

    # Run both tests
    await test_tool_strategy_works()
    await test_provider_strategy_fails()

    # Summary
    print("\n" + "=" * 80)
    print("SUMMARY")
    print("=" * 80)
    print("✅ ToolStrategy works - proves model, schema, and auth are correct")
    print("❌ ProviderStrategy fails with JSONDecodeError at char 0")
    print()
    print("This demonstrates that ProviderStrategy doesn't properly support")
    print("Gemini's native structured output capabilities, despite Gemini")
    print("officially supporting responseSchema.")
    print()
    print("See: https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/control-generated-output")
    print("=" * 80 + "\n")


if __name__ == "__main__":
    asyncio.run(main())

Error Message and Stack Trace (if applicable)

PASSED  [100%]
================================================================================
TEST 2: ProviderStrategy with Gemini
================================================================================
❌ FAILED: StructuredOutputValidationError
Error message: Failed to parse structured output for tool 'response_format': Native structured output expected valid JSON for response_format, but parsing failed: Expecting value: line 1 column 1 (char 0)..

Full traceback:
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1763984729.665989 7897444 fork_posix.cc:71] Other threads are currently calling into gRPC, skipping fork() handlers
Traceback (most recent call last):
  File "/Users/alejandroem/PycharmProjects/dqa/dqa_v3/.venv/lib/python3.13/site-packages/langchain/agents/structured_output.py", line 380, in parse
    data = json.loads(raw_text)
  File "/Users/alejandroem/.local/share/uv/python/cpython-3.13.5-macos-aarch64-none/lib/python3.13/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ~~~~~~~~~~~~~~~~~~~~~~~^^^
  File "/Users/alejandroem/.local/share/uv/python/cpython-3.13.5-macos-aarch64-none/lib/python3.13/json/decoder.py", line 345, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alejandroem/.local/share/uv/python/cpython-3.13.5-macos-aarch64-none/lib/python3.13/json/decoder.py", line 363, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/alejandroem/PycharmProjects/dqa/dqa_v3/.venv/lib/python3.13/site-packages/langchain/agents/factory.py", line 854, in _handle_model_output
    structured_response = provider_strategy_binding.parse(output)
  File "/Users/alejandroem/PycharmProjects/dqa/dqa_v3/.venv/lib/python3.13/site-packages/langchain/agents/structured_output.py", line 387, in parse
    raise ValueError(msg) from e
ValueError: Native structured output expected valid JSON for response_format, but parsing failed: Expecting value: line 1 column 1 (char 0).

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/alejandroem/PycharmProjects/dqa/dqa_v3/reproduce_provider_strategy_bug.py", line 97, in test_provider_strategy_fails
    result = await agent.ainvoke({
             ^^^^^^^^^^^^^^^^^^^^^
        "messages": [{"role": "user", "content": PROMPT}]
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    })
    ^^
  File "/Users/alejandroem/PycharmProjects/dqa/dqa_v3/.venv/lib/python3.13/site-packages/langgraph/pregel/main.py", line 3137, in ainvoke
    async for chunk in self.astream(
    ...<29 lines>...
            chunks.append(chunk)
  File "/Users/alejandroem/PycharmProjects/dqa/dqa_v3/.venv/lib/python3.13/site-packages/langgraph/pregel/main.py", line 2956, in astream
    async for _ in runner.atick(
    ...<13 lines>...
            yield o
  File "/Users/alejandroem/PycharmProjects/dqa/dqa_v3/.venv/lib/python3.13/site-packages/langgraph/pregel/_runner.py", line 304, in atick
    await arun_with_retry(
    ...<15 lines>...
    )
  File "/Users/alejandroem/PycharmProjects/dqa/dqa_v3/.venv/lib/python3.13/site-packages/langgraph/pregel/_retry.py", line 137, in arun_with_retry
    return await task.proc.ainvoke(task.input, config)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alejandroem/PycharmProjects/dqa/dqa_v3/.venv/lib/python3.13/site-packages/langgraph/_internal/_runnable.py", line 705, in ainvoke
    input = await asyncio.create_task(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
        step.ainvoke(input, config, **kwargs), context=context
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/alejandroem/PycharmProjects/dqa/dqa_v3/.venv/lib/python3.13/site-packages/langgraph/_internal/_runnable.py", line 473, in ainvoke
    ret = await self.afunc(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alejandroem/PycharmProjects/dqa/dqa_v3/.venv/lib/python3.13/site-packages/langchain/agents/factory.py", line 1148, in amodel_node
    response = await _execute_model_async(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alejandroem/PycharmProjects/dqa/dqa_v3/.venv/lib/python3.13/site-packages/langchain/agents/factory.py", line 1124, in _execute_model_async
    handled_output = _handle_model_output(output, effective_response_format)
  File "/Users/alejandroem/PycharmProjects/dqa/dqa_v3/.venv/lib/python3.13/site-packages/langchain/agents/factory.py", line 860, in _handle_model_output
    raise validation_error
langchain.agents.structured_output.StructuredOutputValidationError: Failed to parse structured output for tool 'response_format': Native structured output expected valid JSON for response_format, but parsing failed: Expecting value: line 1 column 1 (char 0)..
During task with name 'model' and id '25420273-f168-0d51-97d9-09a38c48e9f8'

Description

Description

Problem Summary

When using ProviderStrategy with Gemini models on Google Cloud VertexAI, LangChain raises a JSONDecodeError: Expecting value: line 1 column 1 (char 0). However, ToolStrategy works perfectly with the exact same configuration.

What I'm trying to do

I want to use ProviderStrategy with Gemini models on VertexAI because:

1. Gemini supports native structured output via the responseSchema parameter (Google Cloud Documentation)
2. ProviderStrategy is more reliable than ToolStrategy per LangChain docs, as it uses provider-native capabilities instead of tool calling
3. Better performance - native JSON output is faster and more reliable than tool-based extraction

What I expect to happen

ProviderStrategy should:

1. Configure the Gemini request with responseMimeType="application/json" and responseSchema=<schema>
2. Receive a properly formatted JSON response
3. Parse it successfully and return structured output

Just like it does for OpenAI's response_format parameter.

What actually happens

ProviderStrategy:

1. Accepts the configuration without complaint
2. Makes a request to Gemini (exact request format unclear)
3. Receives a response that doesn't match expected format
4. Fails with JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Meanwhile, ToolStrategy with the same model/schema/prompt works perfectly, proving the setup is correct.

Why this is a bug in LangChain

Evidence:

1. Gemini officially supports native structured output

   # Official Google GenAI SDK - this works
   from google import genai
   from google.genai import types
   
   response = client.aio.models.generate_content(
       model="gemini-2.5-flash",
       contents=prompt,
       config=types.GenerateContentConfig(
           response_mime_type="application/json",  # ✅ Native support
           response_schema=schema,                  # ✅ Native support
       )
   )

2. ToolStrategy works - proves setup is correct

   • Same model ✅
   • Same schema ✅
   • Same prompt ✅
   • Same credentials ✅
   • Only difference: strategy used

3. ProviderStrategy works for OpenAI but not Gemini

   Provider	Native Support	LangChain Strategy	Status
   OpenAI	✅ response_format	ProviderStrategy	✅ Works
   Gemini	✅ responseSchema	ProviderStrategy	❌ Fails
   Anthropic	❌ None	ToolStrategy	✅ Works

   This inconsistency suggests ProviderStrategy is hardcoded for OpenAI and doesn't properly support Gemini's native structured output API.

4. Error at char 0 suggests response format mismatch

   • The JSONDecodeError at position 0 indicates LangChain received empty or malformed content
   • This suggests ProviderStrategy isn't configuring the Gemini request correctly OR isn't parsing the response correctly

Current workaround

Users must use ToolStrategy (which is what .with_structured_output() uses by default for Gemini):

# Workaround: forced to use ToolStrategy
model = ChatVertexAI(...)
structured_model = model.with_structured_output(MySchema)  # Uses ToolStrategy

This works but defeats the purpose of having native structured output support.

Impact

• Users can't access more reliable ProviderStrategy for Gemini despite the model having native support
• Forced to use slower, less reliable tool calling instead of native capabilities
• Inconsistent behavior - OpenAI gets native support but Gemini doesn't

System Info

System Information

OS: Darwin
OS Version: Darwin Kernel Version 24.6.0: Wed Oct 15 21:12:08 PDT 2025; root:xnu-11417.140.69.703.14~1/RELEASE_ARM64_T6020
Python Version: 3.13.5 (main, Jun 26 2025, 21:28:14) [Clang 20.1.4 ]

Package Information

langchain_core: 1.0.7
langchain: 1.0.8
langchain_community: 1.0.0a1
langsmith: 0.4.38
langchain_aws: 1.0.0
langchain_classic: 1.0.0
langchain_google_vertexai: 3.0.3
langchain_mcp_adapters: 0.1.11
langchain_openai: 1.0.1
langchain_text_splitters: 1.0.0
langgraph_sdk: 0.2.9

Optional packages not installed

langserve

Other Dependencies

aiohttp: 3.13.1
anthropic: 0.54.0
beautifulsoup4: 4.14.2
boto3: 1.40.59
bottleneck: 1.6.0
dataclasses-json: 0.6.7
google-cloud-aiplatform: 1.122.0
google-cloud-storage: 2.19.0
httpx: 0.28.1
httpx-sse: 0.4.3
jsonpatch: 1.33
langgraph: 1.0.3
mcp: 1.19.0
numexpr: 2.14.1
numpy: 2.3.4
openai: 2.6.1
opentelemetry-api: 1.38.0
opentelemetry-exporter-otlp-proto-http: 1.38.0
opentelemetry-sdk: 1.38.0
orjson: 3.11.4
packaging: 25.0
pyarrow: 21.0.0
pydantic: 2.12.3
pydantic-settings: 2.11.0
pytest: 8.4.2
pyyaml: 6.0.3
PyYAML: 6.0.3
requests: 2.32.5
requests-toolbelt: 1.0.0
rich: 14.2.0
sqlalchemy: 2.0.44
SQLAlchemy: 2.0.44
tenacity: 9.1.2
tiktoken: 0.9.0
typing-extensions: 4.15.0
validators: 0.35.0
zstandard: 0.25.0