[Bug] Sampling loop crashes immediately when LLM returns text instead of calling final_response

[strawgate]

Description

run.py:685 raises RuntimeError immediately when structured output is expected (result_type is set) but the LLM returns a text response instead of calling the final_response tool. There is no retry.

This contrasts with validation errors on final_response (lines 657-679), which append an error message to history and retry — giving the LLM a chance to correct itself.

Current behavior

sample_impl (run.py:614)
  → LLM returns text (no tool call)
  → step.is_tool_use = False
  → line 685: raise RuntimeError("Expected structured output of type X, 
      but the LLM returned a text response instead of calling the final_response tool.")

No retry. Immediate crash. The user sees a RuntimeError with no recovery.

When this happens

• Gemini occasionally ignores tool_choice="required" and returns text anyway
• More common with preview models (gemini-3-flash-preview, gemini-3.1-pro-preview)
• Can happen on the 2nd+ iteration of the tool loop when the LLM has gathered information from tools and tries to "answer" with text instead of calling final_response
• Note: Sampling with Tool Calls resets tool_choice #3011 fixed tool_choice resetting to None after the first iteration, which reduced but did not eliminate this issue — some models still return text even with tool_choice="required"

Expected behavior

The sampling loop should retry with an explicit nudge message (similar to how validation errors are handled), capped at a small number of retries (e.g. 2-3) before raising.

Suggested approach:

# At line 682-689, instead of immediate RuntimeError:
if not step.is_tool_use:
    if result_type is not None and result_type is not str:
        text_response_retries += 1
        if text_response_retries > max_text_retries:  # e.g. 3
            raise RuntimeError(...)
        # Append a nudge to history and retry
        step.history.append(SamplingMessage(
            role="user",
            content=TextContent(
                type="text",
                text="You must call the `final_response` tool to provide your answer. "
                     "Do not respond with text — use the tool.",
            ),
        ))
        current_messages = step.history
        continue

Reproduction

import asyncio
from fastmcp import Client, FastMCP, Context
from mcp.types import CreateMessageResult, TextContent
from pydantic import BaseModel

class MyResult(BaseModel):
    answer: str

# Handler that returns text instead of a tool call
def bad_handler(messages, params, context):
    return CreateMessageResult(
        role="assistant",
        model="test",
        content=TextContent(type="text", text="Here is my answer"),
    )

mcp = FastMCP(sampling_handler=bad_handler)

@mcp.tool
async def test_tool(ctx: Context) -> str:
    result = await ctx.sample(messages="Say hello", result_type=MyResult)
    return result.result.answer

async def main():
    async with Client(mcp) as client:
        await client.call_tool("test_tool", {})  # RuntimeError — no retry

asyncio.run(main())

Version

fastmcp 3.2.3

[marvin-context-protocol]

Claude finished @strawgate's task in 3m 17s —— View job

---

Triage: Sampling loop crashes on text response instead of retrying

tl;dr — Confirmed bug. When result_type is set and the LLM returns text instead of calling final_response, sample_impl raises RuntimeError immediately with no retry. The fix is to add a retry-with-nudge loop (matching the existing validation error retry pattern at lines 657–679), capped at ~3 attempts.

• Gather context and read relevant source code
• Analyze the current error handling vs. retry logic
• Verify the bug with a test
• Search for related issues and PRs
• Provide recommendation and action plan

Recommendation

Add a text-response retry mechanism in sample_impl at src/fastmcp/server/sampling/run.py:682-689, matching the pattern already used for validation error retries (lines 657–679). When result_type is set but the LLM returns text:

1. Increment a counter (text_response_retries)
2. If under the retry limit (3), append a user message nudging the LLM to call final_response, and continue
3. If over the limit, raise RuntimeError as today

This is a small, surgical change — ~15 lines of code in one function.

Findings

The asymmetry: sample_impl already handles two kinds of "wrong answer" from the LLM differently:

Scenario	Current behavior	Lines
final_response called with invalid data	Append validation error to history, retry loop continues	657–679
LLM returns text instead of calling final_response	Immediate RuntimeError — no retry	682–689

The validation error path appends a ToolResultContent with isError=True and the validation message, then lets the loop iterate. The text-response path should do something analogous — but since there's no tool call to respond to, a TextContent user message nudge is the right shape.

Why this happens in practice:

• PR Fix tool_choice reset for structured output sampling #3014 (fixing Sampling with Tool Calls resets tool_choice #3011) already ensured tool_choice stays "required" across iterations when result_type is set — see line 700–701
• However, some models (notably Gemini preview models) ignore tool_choice="required" and return text anyway
• This is the remaining gap: even with the correct tool_choice, the loop has no recovery path

Verified by running tests that confirm the immediate crash at run.py:685 — including a scenario where the handler would return the correct final_response on the second call, but never gets the chance.

Verification

I wrote and ran three tests to confirm the behavior:

FAILED test_issue_3847.py::TestIssue3847::test_text_response_crashes_immediately
  → RuntimeError raised immediately (no retry)

FAILED test_issue_3847.py::TestIssue3847::test_text_response_on_second_iteration_crashes
  → Same crash on 2nd iteration after a successful tool call

FAILED test_issue_3847.py::TestIssue3847::test_text_then_correct_tool_call_should_succeed
  → Handler would return correct final_response on 2nd call, but never gets the chance

All three confirm the RuntimeError at run.py:685 with no retry attempt. The error propagates through the server as a ToolError to the client.

Detailed Action Plan

File to modify: src/fastmcp/server/sampling/run.py

Step 1: Add a retry counter (before the loop, around line 595)

max_iterations = 100
max_text_response_retries = 3  # Max retries when LLM returns text instead of final_response
text_response_retries = 0

Step 2: Replace the immediate RuntimeError (lines 682–689)

Current code:

if not step.is_tool_use:
    if result_type is not None and result_type is not str:
        raise RuntimeError(
            f"Expected structured output of type {result_type.__name__}, "
            "but the LLM returned a text response instead of calling "
            "the final_response tool."
        )

Replace with:

if not step.is_tool_use:
    if result_type is not None and result_type is not str:
        text_response_retries += 1
        if text_response_retries > max_text_response_retries:
            raise RuntimeError(
                f"Expected structured output of type {result_type.__name__}, "
                "but the LLM returned a text response instead of calling "
                "the final_response tool."
            )
        # Nudge the LLM to use the final_response tool
        step.history.append(
            SamplingMessage(
                role="user",
                content=TextContent(
                    type="text",
                    text=(
                        "You must call the `final_response` tool to provide "
                        "your answer. Do not respond with text — use the tool."
                    ),
                ),
            )
        )
        current_messages = step.history
        continue

Step 3: Add tests in tests/client/test_sampling_result_types.py

Add two tests to TestSamplingResultType:

1. test_text_response_retries_then_succeeds: Handler returns text on first call, then calls final_response on second call. Assert the result is correct and the handler was called twice.

2. test_text_response_retries_exhausted: Handler always returns text. Assert RuntimeError is raised after retries are exhausted. Verify the handler was called 4 times (1 initial + 3 retries).

3. test_text_response_nudge_message_in_history: Verify that the nudge message appears in the messages sent to the handler on retry, so the LLM gets explicit instructions.

Step 4: Run full test suite and linting

uv sync
uv run pytest -n auto
uv run prek run --all-files

Related Issues and Pull Requests

Repository	Issue or PR	Relevance
PrefectHQ/fastmcp	#3011 — Sampling with Tool Calls resets tool_choice	Predecessor bug — tool_choice was reset to None after first iteration, causing this same crash more frequently. Fixed by #3014, but the immediate-crash behavior remained.
PrefectHQ/fastmcp	#3014 — Fix tool_choice reset for structured output sampling	The fix that kept tool_choice="required" across iterations. Reduced but did not eliminate the text-response crash.

Related Files

File	Relevance	Sections
src/fastmcp/server/sampling/run.py	Contains the bug — sample_impl function with the immediate RuntimeError	L574-703
tests/client/test_sampling_result_types.py	Existing test for validation error retry (test_result_type_validation_error_retries) — the pattern to follow for the new tests	L138-226
src/fastmcp/server/context.py	Context.sample() — the public API that calls sample_impl	L941-1013

---

Marvin Context Protocol | Type /marvin to interact further

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.