feat: tool message conversion and multimodal content support - #115
Conversation
github-actions
Bot
added
documentation
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the Highlights
Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Automated approval for solo maintainer project
This PR has passed all automated quality gates:
- ✅ Static analysis (PHPStan)
- ✅ Code style (PHP-CS-Fixer)
- ✅ Unit & functional tests
- ✅ Security scanning
- ✅ Dependency review
See SECURITY_CONTROLS.md for compensating controls documentation.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #115 +/- ##
============================================
- Coverage 93.89% 93.88% -0.01%
- Complexity 2094 2137 +43
============================================
Files 82 82
Lines 7452 7589 +137
============================================
+ Hits 6997 7125 +128
- Misses 455 464 +9
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces significant enhancements by adding support for multimodal content (images, documents) and standardizing tool message formats across Claude and Gemini providers. The refactoring in ClaudeProvider to centralize message conversion logic is a great improvement for maintainability. The changes are well-supported by a comprehensive set of new unit tests. I have one suggestion for the GeminiProvider to make the tool name resolution more robust and align better with the standard OpenAI message format, avoiding the need for a custom field in tool messages.
There was a problem hiding this comment.
Pull request overview
This PR adds provider-agnostic support for (1) OpenAI-style tool calling message flows and (2) multimodal chat content (text + images/documents), so nr-mcp-agent can interoperate consistently across Claude, Gemini, and OpenAI-compatible backends.
Changes:
- ClaudeProvider: centralizes message conversion (system extraction, tool result/tool call conversion, multimodal content conversion) via
convertMessagesForClaude(). - GeminiProvider: extends message conversion to support tool calling (
functionCall/functionResponse) and multimodal blocks to GeminipartswithinlineData. - Public contract/docs: updates Provider/ToolCapable interface PHPDocs and docs to describe
contentasstring|arrayand adds unit tests for the new behaviors.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
Classes/Provider/ClaudeProvider.php |
Refactors chat/stream payload building to run through a shared Claude message conversion pipeline (tools + multimodal). |
Classes/Provider/GeminiProvider.php |
Adds tool message conversion and multimodal block conversion into Gemini contents/parts request format. |
Classes/Provider/Contract/ProviderInterface.php |
Updates PHPDoc to allow multimodal content arrays. |
Classes/Provider/Contract/ToolCapableInterface.php |
Updates PHPDoc to allow multimodal content arrays for tool-capable chat. |
Documentation/Api/ProviderInterface.rst |
Documents multimodal message content support for chat + tools. |
Tests/Unit/Provider/ClaudeProviderTest.php |
Adds unit tests covering Claude tool conversion and multimodal input handling. |
Tests/Unit/Provider/GeminiProviderTest.php |
Adds unit tests covering Gemini tool conversion and multimodal input handling. |
Tests/Unit/Provider/OpenAiProviderTest.php |
Adds a unit test ensuring OpenAI provider accepts multimodal message content input. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Automated approval for solo maintainer project
This PR has passed all automated quality gates:
- ✅ Static analysis (PHPStan)
- ✅ Code style (PHP-CS-Fixer)
- ✅ Unit & functional tests
- ✅ Security scanning
- ✅ Dependency review
See SECURITY_CONTROLS.md for compensating controls documentation.
bdd0f9a to
66f07a6
Compare
There was a problem hiding this comment.
Automated approval for solo maintainer project
This PR has passed all automated quality gates:
- ✅ Static analysis (PHPStan)
- ✅ Code style (PHP-CS-Fixer)
- ✅ Unit & functional tests
- ✅ Security scanning
- ✅ Dependency review
See SECURITY_CONTROLS.md for compensating controls documentation.
|
Thanks for the thorough reviews @gemini-code-assist @copilot! All findings addressed in
All 2750 unit tests green, PHPStan + CGL passing locally. |
|
@just-tobi I've opened a new pull request, #116, to work on those changes. Once the pull request is ready, I'll request review from you. |
|
Thank you, @just-tobi, for the detailed update and for addressing the findings so thoroughly! It's great to see the improvements, especially the robust handling of Gemini tool name mapping and content normalization. I've noted the changes, and it's good to hear that all tests are passing. I also see that Copilot has opened a new pull request to continue the work. |
There was a problem hiding this comment.
Automated approval for solo maintainer project
This PR has passed all automated quality gates:
- ✅ Static analysis (PHPStan)
- ✅ Code style (PHP-CS-Fixer)
- ✅ Unit & functional tests
- ✅ Security scanning
- ✅ Dependency review
See SECURITY_CONTROLS.md for compensating controls documentation.
Providers now handle content as string (existing) or array of content blocks (new). Each provider converts to its native format: - Claude: convert image_url to image source blocks, document pass-through - Gemini: convert to inlineData parts - OpenAI/Mistral/Groq/OpenRouter/Ollama: pass through (natively supported) Also adds proper tool message conversion for Claude and Gemini providers: - Claude: tool results as tool_result blocks, assistant tool_calls as tool_use - Gemini: tool results as functionResponse, assistant tool_calls as functionCall Fully backward-compatible — string content works unchanged.
028e2e5 to
b2a6c35
Compare
There was a problem hiding this comment.
Automated approval for solo maintainer project
This PR has passed all automated quality gates:
- ✅ Static analysis (PHPStan)
- ✅ Code style (PHP-CS-Fixer)
- ✅ Unit & functional tests
- ✅ Security scanning
- ✅ Dependency review
See SECURITY_CONTROLS.md for compensating controls documentation.
Summary
Two related features that enable nr-mcp-agent to work with all LLM providers for tool calling and file uploads:
1. Tool Message Conversion
role: 'tool'messages to Claudetool_resultcontent blocks, convert assistanttool_callstotool_usecontent blocksrole: 'tool'to GeminifunctionResponseformat, converttool_callstofunctionCallparts2. Multimodal Content Support
chatCompletion()andchatCompletionWithTools()now acceptcontentas array of content blocks (text + images + documents) in addition to plain stringsimage_urlto Claudeimagesource blocks, pass throughdocumentblocksinlineDatapartsWhy
Backward Compatibility
Fully backward-compatible. Existing callers that pass
content: stringare unaffected. The changes are purely additive — providers that receive string content behave exactly as before.Refactoring
convertMessagesForClaude(), shared bychatCompletion(),chatCompletionWithTools(), andstreamChatCompletion()convertToGeminiFormat()withconvertMultimodalToParts()for content block handlingChanges
Classes/Provider/ClaudeProvider.php— refactored with convertMessagesForClaude() + convertMultimodalContent()Classes/Provider/GeminiProvider.php— extended convertToGeminiFormat() + convertMultimodalToParts()Classes/Provider/Contract/ProviderInterface.php— PHPDoc update (content: string|array)Classes/Provider/Contract/ToolCapableInterface.php— PHPDoc updateDocumentation/Api/ProviderInterface.rst— multimodal content documentationTest plan