Skip to content

Cache tool results inside Agent loops (avoid duplicate identical tool calls) #11588

Description

@Aarkin7

Is your feature request related to a problem? Please describe.
Every Agent loop I run for more than ~5 steps re-calls the same tool with the same args. The model fetches a URL, reasons over the result, then fetches the same URL three steps later because the prior answer slipped out of attention. Same shape for retrievers, calculators, web search. ToolInvoker re-runs every call without checking: haystack/components/tools/tool_invoker.py has no cache layer, and haystack/tools/tool.py has no cacheable field. The cost is paid twice each time, the tool's API call plus the LLM tokens spent deciding to make it. On long agent loops this adds up to a real chunk of the run's spend, and I'd rather not pay for the second copy of work the agent already did.

Describe the solution you'd like
A small, opt-in keyed cache around ToolInvoker. Key is (tool_name, sha256(canonicalized_args_json)). Configurable TTL and scope. Per-Tool opt-in so write-effecting tools never serve stale state.

from haystack.tools import Tool, ToolCache, InMemoryToolCache

cache = ToolCache(
    backend=InMemoryToolCache(),
    ttl_seconds=3600,
    scope="agent_run",   # or "session", "global"
)

fetch_url  = Tool(name="fetch_url",  function=requests.get, cacheable=True)
post_slack = Tool(name="post_slack", function=slack_post,    cacheable=False)

agent = Agent(chat_generator=g, tools=[fetch_url, post_slack], tool_cache=cache)

Defaults are chosen to defuse the "stale results" concern up front:

  • cacheable=False on Tool by default. Caching is opt-in per tool, so post_slack and friends can't serve a cached "ok, sent" for a message that was never actually sent.
  • scope="agent_run" by default. No cross-user or cross-tenant leakage unless the user explicitly widens it.
  • Short default TTL (5 minutes). Long enough to deduplicate within an agent loop, short enough that staleness doesn't compound.
  • A ToolCacheStats (hits, misses, calls saved) is appended to the agent's run output, so users can verify the cache is doing something before flipping more tools to cacheable=True.

When no tool_cache is passed, behavior is identical to today.

Describe alternatives you've considered

  • Wrap each tool function in functools.lru_cache. Doesn't compose with Tool.to_dict() serialization, can't be scoped per-Agent, gives no stats in the run output, and every tool author has to remember.
  • LangChain's set_llm_cache or LiteLLM router caching. Different layer. Those cache the LLM call; the duplicate fetch_url still pays the tool's API latency.
  • Make every tool internally idempotent. Doesn't help. The cost being saved isn't the tool's work, it's the agent re-spending tokens to re-decide to call it.
  • A semantic cache over agent steps (cache by input similarity). Bigger scope, contested design, and the wrong layer for the deterministic-args case that bites in practice.

Metadata

Metadata

Assignees

Labels

P3Low priority, leave it in the backlog

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions