Skip to content

response.resume broken when using structured outputs with Anthropic in tool mode #2503

@teamdandelion

Description

@teamdandelion

Description

Consider the following script:

from mirascope import llm


@llm.call("anthropic/claude-sonnet-4-5", format=int)
def lucky_number():
    return "Choose a lucky number between 1 and 10"


result = lucky_number()
second_result = result.resume("Ok, now choose a different lucky number")

This is very simple and should work. However, it raises the following error:

mirascope.llm.exceptions.BadRequestError: Error code: 400 - {'type': 'error', 'error': {'type': 'invalid_request_error', 'message': 'messages.2: `tool_use` ids were found without `tool_result` blocks immediately after: toolu_01NUbw5mvfYRWaDHvMi1xYK1. Each `tool_use` block must have a corresponding `tool_result` block in the next message.'}, 'request_id': 'req_011CXxuFSqH1PCSqbYYrukMX'}

The issue is that:

  1. Anthropic provider uses tool mode by default
  2. Tool mode injects a special MIRASCOPE_FORMAT_TOOL, which is used for the output
  3. When Mirascope processes the assistant message, it converts the tool call into a text block that contains the expected output. There is never a corresponding user block with tool output (because the format tool is never called)
  4. When re-encoding the message history to send to Anthropic, we use the raw representation (unprocessed) which still has a tool call
  5. Anthropic rejects the request because the tool was never called

Solutions that come to mind include:

  1. When encoding the assistant message, check if it contained a format tool invocation, and if so re-generate it to have the text output, not the tool call. Sub-optimal because it may lose reasoning / thinking continuity and (if Anthropic auto-cached its output tokens) invalidate the cache. Also just feels a little hacky.
  2. Inject an additional user message invoking the tool when resuming the conversation. Also feels a little hacky — and in that case will Anthropic reject the subsequent real user message as violating turn order expectations?

Note, this issue is particularly pressing because:

  • Tool use is the default structured format mode for Anthropic
  • Response.validate() is a first class API that wraps a resume loop under the hood

However, we can mitigate the impact of this bug by switching Anthropic to use strict formatting by default (by opting into the beta) for all models that support it.

Thanks to Dahlia-claw at Reverie for finding and reporting this bug.

Python, Mirascope & OS Versions, related packages (not required)

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions