Skip to content

feat(components): support multimodal function tool results - #982

Open
shentongmartin wants to merge 2 commits into
mainfrom
feat/agenticark-multimodal-tool-results
Open

feat(components): support multimodal function tool results#982
shentongmartin wants to merge 2 commits into
mainfrom
feat/agenticark-multimodal-tool-results

Conversation

@shentongmartin

Copy link
Copy Markdown
Contributor

Multimodal Function Tool Results

Problem

FunctionToolResult can represent images in Eino, but the legacy Ark Responses SDK exposes function_call_output.output as a string. Image tool results therefore failed before reaching an Ark vision model.

Solution

Route AgenticArk requests through ark-runtime-go while retaining the existing public configuration and option types. A text-only result remains a string. A result containing an image is sent as an ordered input_text / input_image content array.

Decisions

  • Keep the image attached to function_call_output; do not turn it into an unrelated user message.
  • Reject file:// media inputs before the new SDK can upload local files implicitly.
  • Escape the private bridge marker in plain text and rewrite JSON as raw messages so user text and large integers are preserved exactly.

Key Insight

Tool output modality is part of the conversation protocol. Passing an image as a normal message loses the model's association with the originating tool call; the Responses API must receive it in that call's output item.

Summary

Problem Solution
Ark function tool results accepted only text Send multimodal output through the new Ark runtime SDK
SDK migration could break callers Keep public legacy types and isolate the new transport behind a bridge

多模态函数工具结果

问题

Eino 的 FunctionToolResult 可以表达图片,但旧 Ark Responses SDK 将 function_call_output.output 定义为字符串,导致图片工具结果无法传递给支持视觉能力的 Ark 模型。

方案

内部请求改由 ark-runtime-go 发送,同时保留现有公开配置和 option 类型。纯文本结果仍发送为字符串;包含图片的结果按顺序发送为 input_text / input_image 内容数组。

决策

  • 图片保持在 function_call_output 内,不伪装成无关的 user message。
  • 在新 SDK 自动上传前拒绝 file:// 媒体输入,避免隐式上传本地文件。
  • 对内部 bridge 标记做文本转义,并以 json.RawMessage 改写请求,保证用户文本和大整数不被破坏。

关键洞察

工具输出的模态属于对话协议。将图片作为普通消息传递会丢失其与原工具调用的关联;Responses API 必须在对应工具调用的输出项中接收图片。

总结

问题 方案
Ark 函数工具结果仅支持文本 通过新的 Ark runtime SDK 发送多模态工具输出
SDK 迁移可能破坏调用方 保留公开旧类型,并将新 transport 隔离在 bridge 内

Change-Id: I483e055f3a0a39a72ab8260c5b6740e4b3790a50
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown

Need to create a new tag

The following modules have changes and may need version updates:

  • components/model/agenticark (Current: components/model/agenticark/v0.2.4)

⚠️ Please create and push new version tags for these modules after merging this PR.

Change-Id: I7331e2f23005464df7ab5ee5dbc17297d763c70e
@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown

Need to create a new tag

The following modules have changes and may need version updates:

  • components/model/agenticark (Current: components/model/agenticark/v0.2.4)

⚠️ Please create and push new version tags for these modules after merging this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant