-
Notifications
You must be signed in to change notification settings - Fork 5.2k
Open
Labels
enhancementNew feature or requestNew feature or requesthelp wantedExtra attention is neededExtra attention is neededsize: medium
Description
Problem
@karthikeyavelivela
AI agents interacting with external tools are vulnerable to prompt injection and unintended destructive actions.
Currently, tool execution is dispatched directly through ToolRegistry without a centralized validation layer. This means that if an LLM is manipulated through prompt injection, it could potentially invoke sensitive tools (e.g. destructive GitHub operations, filesystem actions, etc.).
Proposed Solution
Introduce a lightweight security middleware called ToolGuard that validates tool calls before execution.
The middleware intercepts tool calls inside ToolRegistry.get_executor() and enforces configurable policies such as:
- optional allowlist for permitted tools
- destructive tool blocking
- keyword filtering in tool arguments
Design
Implementation points:
- Add a ToolGuard module under core/framework/runner/
- Intercept tool execution in ToolRegistry.get_executor()
- Return structured ToolResult errors when a call is blocked
- Make the policy configurable through AgentRunner
Benefits
- mitigates prompt injection risks
- prevents destructive tool usage
- improves safety for production agents
- keeps enforcement centralized and configurable
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or requesthelp wantedExtra attention is neededExtra attention is neededsize: medium