openaicompat provider silently drops ToolResultOutputContentMedia — image/media tool results are lost

[ezynda3]

Summary

The openaicompat provider (used by OpenRouter, and any OpenAI-compatible API) silently drops tool results that contain media content (ToolResultOutputContentMedia). Only the anthropic provider handles media in tool results.

This means custom tools returning images via ToolOutput.Data/MediaType (which Kit v0.59.0 now correctly converts to ToolResultOutputContentMedia) work with Anthropic but silently produce no tool result message at all for every other provider.

Root cause

In providers/openaicompat/language_model_hooks.go (~line 409), the tool result serialization switch only handles two cases:

switch toolResultPart.Output.GetType() {
case fantasy.ToolResultContentTypeText:
    // ✅ Handled — sends openaisdk.ToolMessage(text, callID)
case fantasy.ToolResultContentTypeError:
    // ✅ Handled — sends openaisdk.ToolMessage(errorText, callID)
// ❌ fantasy.ToolResultContentTypeMedia is completely missing
//    The switch falls through with no match — no message is appended
}

When a tool returns ToolResultOutputContentMedia, the switch has no matching case, so no message is appended to the messages array. The LLM never receives the tool result, which typically causes it to either hallucinate the result or re-call the tool.

How OpenAI API supports this

The OpenAI chat completions API supports multi-content tool messages. A tool result with an image should be serialized as:

{
  "role": "tool",
  "tool_call_id": "call_123",
  "content": [
    {
      "type": "text",
      "text": "Image file: photo.png"
    },
    {
      "type": "image_url",
      "image_url": {
        "url": "data:image/png;base64,iVBOR..."
      }
    }
  ]
}

The openaicompat provider already does this correctly for user messages with FilePart (lines 223-237), where it creates ChatCompletionContentPartImageParam with base64 data URIs. The same pattern should be applied to ToolResultContentTypeMedia.

Comparison with Anthropic provider

The anthropic provider handles all three cases correctly (providers/anthropic/anthropic.go ~line 878):

case fantasy.ToolResultContentTypeMedia:
    content, ok := fantasy.AsToolResultOutputType[fantasy.ToolResultOutputContentMedia](result.Output)
    // Creates an image block alongside optional text
    contentBlocks := []anthropic.ToolResultBlockParamContentUnion{
        {OfImage: anthropic.NewImageBlockBase64(content.MediaType, content.Data).OfImage},
    }
    if content.Text != "" {
        contentBlocks = append(contentBlocks, ...)
    }

Affected providers

Checked all providers in the codebase:

Provider	Handles ToolResultContentTypeMedia?
anthropic	✅ Yes
openaicompat	❌ No — silently dropped
openai	❌ No (uses openaicompat)
openrouter	❌ No (uses openaicompat)
azure	❌ No (uses openaicompat)
google	❌ Not checked, likely missing

Reproduction

tool := kit.NewTool("read_image", "Read an image",
    func(ctx context.Context, input MyInput) (kit.ToolOutput, error) {
        imgBytes, _ := os.ReadFile("photo.png")
        return kit.ToolOutput{
            Content:   "Here is the image",
            Data:      imgBytes,
            MediaType: "image/png",
        }, nil
    },
)

With Anthropic: LLM receives the image and can describe it.
With any OpenAI-compatible provider: LLM receives nothing for the tool call — the entire tool result message is dropped.

Suggested fix

Add a case fantasy.ToolResultContentTypeMedia: to the switch in openaicompat/language_model_hooks.go:

case fantasy.ToolResultContentTypeMedia:
    output, ok := fantasy.AsToolResultOutputType[fantasy.ToolResultOutputContentMedia](toolResultPart.Output)
    if !ok {
        // warn and continue
        continue
    }
    // Build multi-content tool message with image
    var parts []openaisdk.ChatCompletionContentPartUnionParam
    if output.Text != "" {
        parts = append(parts, openaisdk.TextPart(output.Text))
    }
    if strings.HasPrefix(output.MediaType, "image/") {
        dataBytes, err := base64.StdEncoding.DecodeString(output.Data)
        if err == nil {
            dataURI := "data:" + output.MediaType + ";base64," + output.Data
            parts = append(parts, openaisdk.ImagePart(dataURI))
        }
    }
    // Use multi-content tool message
    messages = append(messages, openaisdk.ChatCompletionMessageParamUnion{
        OfTool: &openaisdk.ChatCompletionToolMessageParam{
            ToolCallID: toolResultPart.ToolCallID,
            Content:    openaisdk.ChatCompletionToolMessageParamContentUnion{OfArrayOfContentParts: parts},
        },
    })

The same fix should be applied to the google provider if it has the same gap.

Environment

• Fantasy: v0.17.2
• Kit: v0.59.0 (which correctly sets ToolResponse.Type now)
• Tested with opencode/kimi-k2.5 via OpenRouter

[andreynering]

A pull request for this would be welcome.