Description
What feature would you like to be added?
Tool calling via mechanisms not directly related to autogen studio itself is a must have for any complex agent scenarios. It's currently possible using autogen libraries, but not when using the studio UI or API. In order to rectify this new types should be added to autogen studio to allow for tool calling via external systems, these systems could be, but are not limited to:
- Shell programs
- HTTP servers
- gRPC functions
Above are the primitives, but projects such as model context protocol are trying to solve a similar problem, and may be a good option for this kind of capability.
A community extension already exists which uses MCP with autogen: https://github.com/richard-gyiko/autogen-ext-mcp
Based on the current implementation of tool calling I believe a 2 phased approach would be best for solving this problem.
Phase 1:
The first step to remote tool calling would be to introduce remote tool call options within the existing Component
framework found in component_factory.py
. There are 2 issues with the current tool definition:
- The definition is not a
Union
like other types. This should be changed to something like the following:
class BaseToolConfig(BaseConfig):
name: str
description: str
tool_type: ToolTypes
component_type: ComponentTypes = ComponentTypes.TOOL
class PythonFunctionToolConfig(BaseToolConfig):
content: str
tool_type: ToolTypes = ToolTypes.PYTHON_FUNCTION
class McpToolConfig(BaseToolConfig):
tool_type: ToolTypes = ToolTypes.MCP_CLIENT
ToolConfig = PythonFunctionToolConfig | McpToolConfig
- Each tool
Component
can only load a singleTool
. This can become a problem because individual remote locations may host many tools. This leads into Phase 2
Phase 2:
The 2nd phase would be to introduce an entirely new type to the schema which may be called something like ToolCollection
. A ToolCollection
would be an endpoint or other location which may have many tools available, and the user may want to include many tools from said location under one "connection" rather than needing to include the same location many times.
My code designs for this step are not nearly as fleshed out as they will require larger changes to the data model.
Why is this needed?
Currently tool calling in autogen studio is very limited because the only way to add custom tools is via inline Python
which has quite a few issues:
- Testing is hard
- The form factor is ugly
- Any use cases other than very simple because extremely difficult, especially when a lot of dependencies are required.