Skip to content

External tool calling API for autogen studio #5170

Closed
@EItanya

Description

@EItanya

What feature would you like to be added?

Tool calling via mechanisms not directly related to autogen studio itself is a must have for any complex agent scenarios. It's currently possible using autogen libraries, but not when using the studio UI or API. In order to rectify this new types should be added to autogen studio to allow for tool calling via external systems, these systems could be, but are not limited to:

  1. Shell programs
  2. HTTP servers
  3. gRPC functions

Above are the primitives, but projects such as model context protocol are trying to solve a similar problem, and may be a good option for this kind of capability.

A community extension already exists which uses MCP with autogen: https://github.com/richard-gyiko/autogen-ext-mcp

Based on the current implementation of tool calling I believe a 2 phased approach would be best for solving this problem.

Phase 1:

The first step to remote tool calling would be to introduce remote tool call options within the existing Component framework found in component_factory.py. There are 2 issues with the current tool definition:

  1. The definition is not a Union like other types. This should be changed to something like the following:
class BaseToolConfig(BaseConfig):
    name: str
    description: str
    tool_type: ToolTypes
    component_type: ComponentTypes = ComponentTypes.TOOL


class PythonFunctionToolConfig(BaseToolConfig):
    content: str
    tool_type: ToolTypes = ToolTypes.PYTHON_FUNCTION


class McpToolConfig(BaseToolConfig):
    tool_type: ToolTypes = ToolTypes.MCP_CLIENT


ToolConfig = PythonFunctionToolConfig | McpToolConfig
  1. Each tool Component can only load a single Tool. This can become a problem because individual remote locations may host many tools. This leads into Phase 2

Phase 2:

The 2nd phase would be to introduce an entirely new type to the schema which may be called something like ToolCollection. A ToolCollection would be an endpoint or other location which may have many tools available, and the user may want to include many tools from said location under one "connection" rather than needing to include the same location many times.

My code designs for this step are not nearly as fleshed out as they will require larger changes to the data model.

Why is this needed?

Currently tool calling in autogen studio is very limited because the only way to add custom tools is via inline Python which has quite a few issues:

  1. Testing is hard
  2. The form factor is ugly
  3. Any use cases other than very simple because extremely difficult, especially when a lot of dependencies are required.

Metadata

Metadata

Assignees

No one assigned

    Labels

    proj-studioRelated to AutoGen Studio.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions