External tool calling API for autogen studio

[EItanya]

What feature would you like to be added?

Tool calling via mechanisms not directly related to autogen studio itself is a must have for any complex agent scenarios. It's currently possible using autogen libraries, but not when using the studio UI or API. In order to rectify this new types should be added to autogen studio to allow for tool calling via external systems, these systems could be, but are not limited to:

1. Shell programs
2. HTTP servers
3. gRPC functions

Above are the primitives, but projects such as model context protocol are trying to solve a similar problem, and may be a good option for this kind of capability.

A community extension already exists which uses MCP with autogen: https://github.com/richard-gyiko/autogen-ext-mcp

Based on the current implementation of tool calling I believe a 2 phased approach would be best for solving this problem.

Phase 1:

The first step to remote tool calling would be to introduce remote tool call options within the existing Component framework found in component_factory.py. There are 2 issues with the current tool definition:

1. The definition is not a Union like other types. This should be changed to something like the following:

class BaseToolConfig(BaseConfig):
    name: str
    description: str
    tool_type: ToolTypes
    component_type: ComponentTypes = ComponentTypes.TOOL


class PythonFunctionToolConfig(BaseToolConfig):
    content: str
    tool_type: ToolTypes = ToolTypes.PYTHON_FUNCTION


class McpToolConfig(BaseToolConfig):
    tool_type: ToolTypes = ToolTypes.MCP_CLIENT


ToolConfig = PythonFunctionToolConfig | McpToolConfig

2. Each tool Component can only load a single Tool. This can become a problem because individual remote locations may host many tools. This leads into Phase 2

Phase 2:

The 2nd phase would be to introduce an entirely new type to the schema which may be called something like ToolCollection. A ToolCollection would be an endpoint or other location which may have many tools available, and the user may want to include many tools from said location under one "connection" rather than needing to include the same location many times.

My code designs for this step are not nearly as fleshed out as they will require larger changes to the data model.

Why is this needed?

Currently tool calling in autogen studio is very limited because the only way to add custom tools is via inline Python which has quite a few issues:

1. Testing is hard
2. The form factor is ugly
3. Any use cases other than very simple because extremely difficult, especially when a lot of dependencies are required.

[victordibia]

Hi @EItanya ,

Thanks so much for this indepth review.
I would really love to work with you on making these exact list of functionality available. There is strong alignment with the goals we have for AGS.

Here are two throughts.

Current DSL for AGS is Limited (and UGLY)

The current version of AGS implements its own schema (the db types that you reference and component_factory). In addition to being limited (as you correctly point out), it is also an additional layer of indirection (i.e., it might load/serialize components in ways that differ from what the low level library does).

The solution? Have components in AutoGen have an interface to self serialize/load. And AGS should use this natively. This means you can build whatever you want in python and call component.dump_component() and just use that in AGS.

We have been working on that here #4439.

Remote Tools

I think we could create additional tool classes that inherit from BaseTool, like FunctionTool but with additional behaviours e.g,. APITool, ShellTool, MCPTool etc.
These tools should be made declarative see #5036 and #5052

Once this is complete, it should be possible to just use them in AGS directly #5172.

Phase 2 sounds good (tool collections).

What do you think?
Once we agree, we can collaborate on a plan of action.

[EItanya]

Honestly I think I agree on all points 😆

The thing we want to build aligns very very closely with the apiserver that autogen-studio implements, minus the above features you mentioned so would definitely love to work closely to make those available.

As I mentioned I actually have a working prototype with MCP, but I think I found some bugs in the upstream mcp sdk which would definitely need to be worked out before anything can be merged, but I will put up a draft later today as a proof of concept to spur further discussion.

One thing you mentioned is of particular interest to me, specifically: The solution? Have components in AutoGen have an interface to self serialize/load. And AGS should use this natively. This means you can build whatever you want in python and call component.dump_component() and just use that in AGS.. I am going to read through those issues and PRs to get a better understanding of the path and then we can link up and decide how I can help.

[victordibia]

Sounds good, I look forward to exploring your PR at some point. I’ll also tag you for info only on relevant PRs above as they get merged.

[victordibia]

Hi @EItanya ,

Now that we have AGS replatformed on the component config interface, and we have httptools, and mcp tools .. perhaps we can consider this issue addressed?

The one thing we have not explicitly addressed is the idea of a "tool collection".
Perhaps this is something that the MCP use case addresses?

[EItanya]

I buy that argument. I do think the Tool Collection is still a concept missing from autogenstudio if run standalone, but they are now doable with some custom components.

I don't feel too strongly, so feel free to close this issue if you see it as completed!