Is your feature request related to a problem? Please describe.
Every Agent loop I run for more than ~5 steps re-calls the same tool with the same args. The model fetches a URL, reasons over the result, then fetches the same URL three steps later because the prior answer slipped out of attention. Same shape for retrievers, calculators, web search. ToolInvoker re-runs every call without checking: haystack/components/tools/tool_invoker.py has no cache layer, and haystack/tools/tool.py has no cacheable field. The cost is paid twice each time, the tool's API call plus the LLM tokens spent deciding to make it. On long agent loops this adds up to a real chunk of the run's spend, and I'd rather not pay for the second copy of work the agent already did.
Describe the solution you'd like
A small, opt-in keyed cache around ToolInvoker. Key is (tool_name, sha256(canonicalized_args_json)). Configurable TTL and scope. Per-Tool opt-in so write-effecting tools never serve stale state.
from haystack.tools import Tool, ToolCache, InMemoryToolCache
cache = ToolCache(
backend=InMemoryToolCache(),
ttl_seconds=3600,
scope="agent_run", # or "session", "global"
)
fetch_url = Tool(name="fetch_url", function=requests.get, cacheable=True)
post_slack = Tool(name="post_slack", function=slack_post, cacheable=False)
agent = Agent(chat_generator=g, tools=[fetch_url, post_slack], tool_cache=cache)
Defaults are chosen to defuse the "stale results" concern up front:
cacheable=False on Tool by default. Caching is opt-in per tool, so post_slack and friends can't serve a cached "ok, sent" for a message that was never actually sent.
scope="agent_run" by default. No cross-user or cross-tenant leakage unless the user explicitly widens it.
- Short default TTL (5 minutes). Long enough to deduplicate within an agent loop, short enough that staleness doesn't compound.
- A
ToolCacheStats (hits, misses, calls saved) is appended to the agent's run output, so users can verify the cache is doing something before flipping more tools to cacheable=True.
When no tool_cache is passed, behavior is identical to today.
Describe alternatives you've considered
- Wrap each tool function in
functools.lru_cache. Doesn't compose with Tool.to_dict() serialization, can't be scoped per-Agent, gives no stats in the run output, and every tool author has to remember.
- LangChain's
set_llm_cache or LiteLLM router caching. Different layer. Those cache the LLM call; the duplicate fetch_url still pays the tool's API latency.
- Make every tool internally idempotent. Doesn't help. The cost being saved isn't the tool's work, it's the agent re-spending tokens to re-decide to call it.
- A semantic cache over agent steps (cache by input similarity). Bigger scope, contested design, and the wrong layer for the deterministic-args case that bites in practice.
Is your feature request related to a problem? Please describe.
Every Agent loop I run for more than ~5 steps re-calls the same tool with the same args. The model fetches a URL, reasons over the result, then fetches the same URL three steps later because the prior answer slipped out of attention. Same shape for retrievers, calculators, web search.
ToolInvokerre-runs every call without checking:haystack/components/tools/tool_invoker.pyhas no cache layer, andhaystack/tools/tool.pyhas nocacheablefield. The cost is paid twice each time, the tool's API call plus the LLM tokens spent deciding to make it. On long agent loops this adds up to a real chunk of the run's spend, and I'd rather not pay for the second copy of work the agent already did.Describe the solution you'd like
A small, opt-in keyed cache around
ToolInvoker. Key is(tool_name, sha256(canonicalized_args_json)). Configurable TTL and scope. Per-Tool opt-in so write-effecting tools never serve stale state.Defaults are chosen to defuse the "stale results" concern up front:
cacheable=FalseonToolby default. Caching is opt-in per tool, sopost_slackand friends can't serve a cached"ok, sent"for a message that was never actually sent.scope="agent_run"by default. No cross-user or cross-tenant leakage unless the user explicitly widens it.ToolCacheStats(hits, misses, calls saved) is appended to the agent's run output, so users can verify the cache is doing something before flipping more tools tocacheable=True.When no
tool_cacheis passed, behavior is identical to today.Describe alternatives you've considered
functools.lru_cache. Doesn't compose withTool.to_dict()serialization, can't be scoped per-Agent, gives no stats in the run output, and every tool author has to remember.set_llm_cacheor LiteLLM router caching. Different layer. Those cache the LLM call; the duplicatefetch_urlstill pays the tool's API latency.