[EPIC] Create MCP proxy from OpenAPI specifications

[davorrunje]

Description:
We need a tool that automatically generates an stdio-based MCP server from an OpenAPI specification (either through a URL or as a JSON file). This server will connect to the service described in the OpenAPI spec and expose it as an MCP tool. The main goal is to enable easy creation of authenticated MCP tools for any service that has an OpenAPI specification, including the handling of authentication and security mechanisms as specified by the OpenAPI standard.

Key Features & Requirements:

• Input:

  • The tool should accept either a URL or a JSON file as input, containing the OpenAPI specification.
  • A path to a folder where to export generated files

• Output:

  • Files generated in the given folder.
  • Generate a stdio-based MCP server that translates the API described in the OpenAPI spec into an MCP tool.
  • The server should support the API operations defined in the OpenAPI specification and provide a standard interface for interacting with them via MCP.

• Authentication and Security:

  • The tool should handle the authentication and security mechanisms defined in the OpenAPI specification (e.g., OAuth2, API keys).
  • Secrets (API keys, OAuth credentials, etc.) will be passed to the stdio-based MCP server via environment variables. The tool must support securely retrieving these secrets from environment variables, in alignment with the OpenAPI security schemes (such as API keys or OAuth tokens).
  • The generated MCP server should map the authentication methods described in the OpenAPI spec to the standard ways supported by the MCP specification, using these environment variables.

• MCP Server:

  • The generated MCP server should run over stdio (standard input and output) to interact with external services via a text-based interface.
  • The server must communicate with the service described in the OpenAPI specification using the methods and authentication schemes defined in the specification, with secrets securely passed via environment variables.

• Error Handling:

  • The generated MCP tool should provide robust error handling and clear error messages when interacting with the service.
  • Specifically, errors related to authentication failures (e.g., missing or invalid environment variables) should be clearly indicated.

• Environment Variables:

  • The tool must allow users to set relevant secrets (such as API keys or OAuth tokens) as environment variables.
  • The tool should read these environment variables to authenticate requests and interact with the service described by the OpenAPI specification.

• Use Case:

  • This tool is meant to allow easy conversion of OpenAPI specifications into an MCP server, enabling any API described by an OpenAPI document to be directly usable as an MCP tool.
  • The use of stdio allows the generated MCP tool to interact seamlessly with other applications, scripts, and agents that support the MCP format.
  • The use of environment variables for secrets adds a layer of security and ensures best practices for managing sensitive data.

Motivation:

• OpenAPI specifications are widely available for modern APIs, and this tool would automate the process of creating an MCP server that can serve those APIs in a standardized way.
• By handling authentication and security according to the OpenAPI spec, and by using environment variables for secrets, it ensures that the resulting MCP tools follow common and widely accepted practices, making it easy to integrate and maintain them in enterprise environments.

Acceptance Criteria:

• A working tool that takes an OpenAPI URL or JSON as input.
• The tool generates an MCP server running over stdio that:
  • Supports all API operations defined in the OpenAPI spec.
  • Handles authentication and security mechanisms as defined in the spec (e.g., OAuth2, API keys).
  • Accepts secrets through environment variables for authentication.
  • Outputs responses in the MCP format, adhering to the protocol’s requirements.
  • Provides appropriate error handling for issues such as missing or invalid environment variables for authentication.

Example Code:

The following Python example demonstrates how to generate an MCP server from an OpenAPI specification, authenticate using environment variables, and interact with the API through an LLM agent:

from pathlib import Path
from mcp import ClientSession, StdioServerParameters
from mcp.client.sse import sse_client
from mcp.client.stdio import stdio_client
from autogen import LLMConfig
from autogen.agentchat import AssistantAgent
from autogen.mcp import create_toolkit
from autogen.mcp.mcp_proxy import MCPProxy

client_source_path = Path(...)

# Create an MCP Proxy server based on the OpenAPI specification
MCPProxy.create(
    openapi_url="https://dev.infobip.com/openapi/products/whatsapp.json",  # OpenAPI spec URL
    client_source_path=client_source_path,  # Path to the generated client source
    servers=[{"url": "https://api.infobip.com"}],  # List of servers to connect to
)

async def create_toolkit_and_run(session: ClientSession) -> None:
    # Create a toolkit with available MCP tools
    toolkit = await create_toolkit(session=session)
    
    # Initialize an AssistantAgent with GPT-4o-mini model
    agent = AssistantAgent(name="assistant", llm_config=LLMConfig(model="gpt-4o-mini", api_type="openai"))
    
    # Register the toolkit with the agent
    toolkit.register_for_llm(agent)

    # Make a request using the MCP tool
    result = await agent.a_run(
        message="Add 123223 and 456789",  # Example message for the tool
        tools=toolkit.tools,
        max_turns=2,
        user_input=False,
    )
    await result.process()

# Create server parameters for stdio connection, including environment variables for secrets
server_params = StdioServerParameters(
    command="python",  # The command to run the server
    args=[
        str(tmp_path / "main_tmp.py"),  # Path to the server script
        "stdio",  # Stdio transport mode
    ],
    env={
        "API_KEY": "your-api-key",  # Example API Key passed as an environment variable
        "OAUTH2_TOKEN": "your-oauth2-token",  # Example OAuth2 token passed as an environment variable
    },
)

# Initialize the stdio connection and create the MCP server session
async with stdio_client(server_params) as (read, write), ClientSession(read, write) as session:
    await session.initialize()  # Initialize the session
    await create_toolkit_and_run(session)  # Run the toolkit and agent

Explanation:

1. MCPProxy.create():

   • This method creates an MCP proxy server based on an OpenAPI specification, connecting to the service described in the spec.
   • You need to provide the OpenAPI URL and the client source path.

2. create_toolkit_and_run():

   • This function creates a toolkit using the MCP session and registers an LLM agent (AssistantAgent), which can use the MCP tools.
   • The agent can make requests using the available tools.

3. Environment Variables:

   • Authentication secrets like API_KEY and OAUTH2_TOKEN are passed as environment variables to the stdio_client. The server uses these variables to authenticate with the service.

4. Stdio Connection:

   • The connection is established using stdio_client, which communicates with the server through standard input and output.
   • The ClientSession manages the session between the MCP server and the agent.

Additional Notes:

• The tool should clearly document which environment variables are required (e.g., API_KEY, OAUTH2_TOKEN).
• Consider the inclusion of any necessary libraries for API requests, such as requests or httpx, to facilitate communication with the service described in the OpenAPI specification.
• The generated code should be easily extendable to handle more complex use cases or API specifications.