feat: CodeMode sandbox tool for custom one-off tools, and multi-step tool chaining

[kingpanther13]

Summary

Add an optional sandboxed code execution tool that lets LLMs create custom one-off tools on the fly when no existing tool covers the user's request, using FastMCP 3.1's CodeMode sandbox (pydantic-monty).

Problem

The ha-mcp tool catalog — no matter how large — will never cover every possible Home Assistant operation. Today, when a user asks for something there's no dedicated tool for (e.g., "edit entity categories", "pull debug logs", "interact with a third-party add-on"), the LLM fails in one of several ways:

• Gives up entirely: "Sorry, I don't have a tool for that" — even though the operation is technically possible through existing lower-level APIs.
• Confabulates: Claims the operation is impossible or unsupported by Home Assistant, when it's really just unsupported by the current tool catalog.
• Thrashes through similar tools: Tries a series of related-but-wrong tools, failing repeatedly, wasting tokens and round-trips before ultimately giving up anyway.

But the LLM could accomplish many of these tasks by writing custom code that calls the existing lower-level tools or HA APIs in the right combination. The gap isn't always capability — it's that the LLM has no way to run custom code to bridge the gaps.

Proposal

Add an ha_execute_code tool that accepts Python code with access to call_tool(name, args). The code runs in Monty — a sandboxed Python interpreter (Rust-based, no filesystem/network access, configurable resource limits).

Primary use case: On-the-fly custom tools

The core value is giving the LLM a general-purpose fallback. When no existing tool matches the user's request, the LLM writes custom Python code on the spot to accomplish it:

User: "Move my kitchen sensors to the 'Climate' category"
LLM:  No dedicated tool exists for editing entity categories.
      → Writes Python that calls the HA REST API via available tools to modify categories
      → Returns the result

User: "Show me the debug logs for the Zigbee integration"
LLM:  No tool exposes debug logs directly.
      → Writes Python to access the debug log endpoint and filter for the relevant integration
      → Returns the result

User: "Check the status of my AdGuard add-on"
LLM:  No tool covers third-party add-on interaction.
      → Writes Python to query the Supervisor add-on API
      → Returns the result

The code isn't persistent — it runs once, returns a result, and is gone. The LLM essentially gains the ability to improvise rather than being limited to a fixed set of pre-built tools. This dramatically expands what ha-mcp can handle without needing to anticipate and build a dedicated tool for every possible request.

Secondary use case: Multi-step chaining

As a bonus, the sandbox also lets LLMs chain multiple tool calls in a single round-trip (e.g., "find all lights on for more than 2 hours and turn them off") without burning tokens on intermediate results flowing back through the context window.

Two implementation options:

Option A: Custom standalone tool (simpler)

Register ha_execute_code as a regular MCP tool using pydantic-monty directly (already a dependency via fastmcp[code-mode]). Can be pinned or placed behind a write proxy. No transform needed.

@mcp.tool(annotations=ToolAnnotations(destructiveHint=True))
async def ha_execute_code(code: str, ctx: Context) -> Any:
    """Run sandboxed Python that can chain tool calls via call_tool(name, args)."""
    async def call_tool(tool_name: str, arguments: dict) -> Any:
        return await ctx.fastmcp.call_tool(tool_name, arguments)

    m = pydantic_monty.Monty(code, external_functions=['call_tool'])
    return await run_monty_async(m,
        external_functions={'call_tool': call_tool},
        limits=ResourceLimits(max_duration_secs=30.0, max_memory=10*1024*1024))

Option B: Separate CodeMode provider (more integrated)

Mount a sub-server with the CodeMode transform that runs concurrently with the main tool system. Provides the full CodeMode discovery flow (search → get_schema → execute) scoped to its own provider.

Sandbox constraints (security features)

• No class definitions, no match statements
• No third-party imports (only sys, os, typing, asyncio, re)
• No filesystem or network access — all I/O through call_tool() only
• Configurable limits: time, memory, allocations, recursion depth

Configuration

• Toggle via config: ENABLE_CODE_MODE: bool = False
• Resource limits configurable via settings

Gotchas & Guardrails

This tool is powerful but comes with a significant risk: LLMs may reach for it when they shouldn't.

• Lazy fallback: If the LLM can't immediately find the right tool, it may jump straight to ha_execute_code instead of trying harder to discover the correct existing tool. The hand-written code will almost always do a worse job than a purpose-built tool would have.
• Hallucinated necessity: The LLM may convince itself that a custom tool is needed when a perfectly good existing tool is available — it just didn't search for it properly, or misunderstood what an existing tool can do.
• Quality gap: Purpose-built tools have proper error handling, validation, and tested behavior. One-off code written by an LLM in a sandbox will be more fragile and less reliable.

Guardrails to consider:

• Mark the tool as destructiveHint=True so MCP clients that respect annotations will prompt the user before execution.
• The tool description should explicitly instruct the LLM to only use this as a last resort after confirming no existing tool can accomplish the task.
• Consider requiring the LLM to provide a justification parameter explaining why no existing tool suffices, which could be shown to the user for approval.
• Consider rate limiting or requiring user confirmation before each execution.
• Logging/auditing of all code executed through this tool is essential.

Context

This pairs well with the search-based tool discovery system being explored. The code sandbox gives LLMs an escape hatch for requests that fall outside the tool catalog, while the search system handles discovery of tools that do exist.

References

• FastMCP 3.1 CodeMode docs
• pydantic-monty (the sandbox runtime)
• FastMCP 3.1 release notes

[github-actions]

🔍 Analysis

The issue proposes adding a sandboxed code execution tool, ha_execute_code, to enable LLMs to chain multiple Home Assistant tool calls in a single round-trip. This would significantly improve token efficiency and performance for complex, multi-step operations (e.g., "find all lights on for >2 hours and turn them off").

This feature leverages FastMCP 3.1's CodeMode and the Monty sandbox (a Rust-based Python interpreter). Since the project already uses fastmcp==3.1.0, the infrastructure is ready, although pydantic-monty must be added as a dependency. The proposal includes strong security constraints (no filesystem/network access, resource limits) which are essential for this kind of "escape hatch" tool.

The ha_execute_code tool does not currently exist in ha-mcp. Its implementation would be a valuable addition for advanced automation tasks that are currently slow or expensive due to multiple LLM round-trips.

🛠️ Suggested Steps

1. Add Dependencies: Update pyproject.toml to include pydantic-monty in the dependencies list.
2. Update Configuration: Add ENABLE_CODE_MODE: bool = False to the Settings class in src/ha_mcp/config.py to allow users to opt-in to this feature.
3. Implement the Tool: Create a new module src/ha_mcp/tools/tools_code.py and implement the ha_execute_code tool using the pydantic-monty runtime as suggested in the issue.
4. Register the Module: The ToolsRegistry will automatically discover and register the new module if it follows the tools_*.py naming convention and contains a register_code_tools(mcp, client, **kwargs) function.
5. Refine Sandbox Limits: Ensure the default resource limits (30s duration, 10MB memory) are configurable via the Settings class.

📊 Technical Analysis (for maintainers)

Root Cause

Current multi-step operations require the LLM to process results and issue new tool calls one by one, consuming context window and time. There is no existing way to execute "macro" logic locally on the server side within a single MCP tool call.

Affected Code

• pyproject.toml - Needs pydantic-monty added to dependencies.
• src/ha_mcp/config.py - Needs ENABLE_CODE_MODE and associated resource limit settings.
• src/ha_mcp/server.py - Needs logic to optionally add the CodeMode transform if Option B is chosen.
• src/ha_mcp/tools/tools_code.py - New file for the tool implementation.

Proposed Fix

Create src/ha_mcp/tools/tools_code.py:

import logging
from typing import Any
from fastmcp import Context
from .helpers import log_tool_usage

logger = logging.getLogger(__name__)

def register_code_tools(mcp, client, **kwargs):
    settings = kwargs.get("server").settings
    if not settings.enable_code_mode:
        return

    try:
        import pydantic_monty
        from pydantic_monty import ResourceLimits
    except ImportError:
        logger.warning("pydantic-monty not installed, ha_execute_code will be unavailable")
        return

    @mcp.tool(annotations={"destructiveHint": True, "title": "Execute Code"})
    @log_tool_usage
    async def ha_execute_code(code: str, ctx: Context) -> Any:
        """Run sandboxed Python that can chain tool calls via call_tool(name, args)."""
        async def call_tool(tool_name: str, arguments: dict) -> Any:
            return await ctx.fastmcp.call_tool(tool_name, arguments)

        m = pydantic_monty.Monty(code, external_functions=['call_tool'])
        return await pydantic_monty.run_monty_async(
            m,
            external_functions={'call_tool': call_tool},
            limits=ResourceLimits(
                max_duration_secs=settings.code_mode_max_duration, 
                max_memory=settings.code_mode_max_memory
            )
        )

Related Issues/PRs

No similar issues found.

---

🤖 Automated analysis by Issue Bot

[kingpanther13]

My thought process on this was less about the multi tool chaining and more about using code mode to make custom tools on the fly, so I adjusted the issue description to reflect that. Might be a great way to not have to make more tools.... BUT definitely needs guardrails and needs a lot of warnings and whatnot because it will easily be able to cause a lot of damage if left unchecked.

[m13v]

This is a really interesting approach. Being able to generate one-off tools at runtime covers the long tail of home automation requests that no fixed tool set can anticipate.

A few considerations from building tool-heavy MCP servers:

1. Security boundary - the sandbox needs to be tight. Home Assistant controls physical devices, so a generated tool that accidentally toggles a lock or garage door could be dangerous. Consider a whitelist of allowed HA API endpoints that generated tools can call, rather than giving full access.

2. Tool chaining - for multi-step operations ("turn off all lights except the bedroom, then set the thermostat"), you want the LLM to be able to compose tools rather than generating one mega-tool. A simple pipeline abstraction where tool N's output feeds tool N+1's input would be cleaner.

3. Caching generated tools - if a user asks for the same custom operation repeatedly ("dim the living room to movie mode"), cache the generated tool code so it does not need to be re-synthesized each time. This also lets users audit and approve the generated code once.

4. MCP tool annotations - mark generated tools with destructiveHint: true by default so the client can prompt for confirmation before executing them.

[m13v]

For reference, our macOS automation MCP server handles both read-only and destructive tool calls with proper annotations: https://github.com/mediar-ai/mcp-server-macos-use/blob/main/Sources/MCPServer/main.swift

The desktop automation engine that powers it: https://github.com/mediar-ai/terminator/blob/main/crates/terminator/src/element.rs

More context: https://fazm.ai/gh