Skip to content

Commit 279c57f

Browse files
joaomdmouraclaude
andcommitted
feat(tools): surface tool failures instead of reporting them as success
A tool can finish without raising and still fail to do what it was asked. Slack answers HTTP 200 with `{"ok": false, "error": "channel_not_found"}`; an MCP server sets `isError`; a CrewAI AMP action returns `API request failed: ...`. In every case the call "worked", so the error text reached the agent as an ordinary result, the agent narrated the problem in prose, and the run was recorded as a success. Concretely: five failed `slackbot_send_message` calls each rendered as "Tool Execution Completed", the task passed, and the crew passed -- with the only evidence being a sentence in the final answer. Nothing downstream could tell the difference, and an agent that keeps going on a step that silently did nothing builds the rest of its work on it. Give that outcome a type and a reaction: - `ToolFailure` -- what a tool returns instead of an error string. The agent still reads prose via `as_agent_message()`, so model behavior is unchanged; the framework now knows the call failed. - `ToolFailurePolicy` -- `ignore` (previous behavior), `warn` (default: record + emit, keep going), `raise` (abort with `ToolExecutionFailedError`). Resolved most-specific-first: tool, task, agent, crew. - `ToolFailureDetectedEvent` -- emitted before a `raise` aborts, so subscribers always observe the failure. `ToolUsageFinishedEvent` also carries a `failure` field so a trace UI can mark the call failed without correlating two events. - `tool_failures` on `TaskOutput`, `CrewOutput` and `LiteAgentOutput`, plus `has_tool_failures`, so consumers never parse a string. Detection is strictly declarative -- no string sniffing, so a tool that legitimately returns text about an error is never misread as failing. Failures come from a returned `ToolFailure`, a raised exception, MCP `isError`, a spent `max_usage_count`, or an unknown tool. Wired into all four tool-execution paths (the ReAct path and the three native function-calling implementations). Sources updated to report structurally: `MCPClient.call_tool_result()` preserves `isError` that `call_tool()` dropped, and `CrewAIPlatformActionTool` returns a `ToolFailure` for non-2xx and for caught exceptions. Two latent bugs fixed along the way: `ToolUsage` assumed every agent has a `fingerprint` (LiteAgent does not), and policy resolution now tolerates malformed values rather than letting telemetry take down a tool call. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ETacm2dMASfpMAYUiDu5YG
1 parent f15844b commit 279c57f

28 files changed

Lines changed: 1346 additions & 21 deletions

File tree

docs/edge/en/concepts/tools.mdx

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -334,6 +334,118 @@ writer1 = Agent(
334334
#...
335335
```
336336

337+
## Reporting Tool Failures
338+
339+
A tool can finish without raising and still fail to do what it was asked. Slack
340+
answers `HTTP 200` with `{"ok": false, "error": "channel_not_found"}`; an MCP
341+
server sets `isError`; a platform action returns an error payload. The tool call
342+
"worked", so the error text reaches the agent as an ordinary result — the agent
343+
narrates the problem in its final answer and the run is recorded as a success.
344+
345+
Return a `ToolFailure` instead of an error string and the framework can tell the
346+
difference:
347+
348+
```python Code
349+
from typing import Any
350+
351+
from crewai.tools import BaseTool
352+
from crewai.tools.tool_failure import ToolFailure
353+
354+
355+
class SendSlackMessage(BaseTool):
356+
name: str = "send_slack_message"
357+
description: str = "Post a message to a Slack channel."
358+
359+
def _run(self, channel: str, text: str) -> Any:
360+
payload = slack.post(channel=channel, text=text)
361+
if not payload["ok"]:
362+
return ToolFailure(
363+
message=f"Slack rejected the message: {payload['error']}",
364+
code=payload["error"],
365+
retryable=payload["error"] == "rate_limited",
366+
)
367+
return payload
368+
```
369+
370+
The agent still reads plain prose — `ToolFailure.as_agent_message()` renders the
371+
message — so model behavior is unchanged. What changes is that the failure is now
372+
visible to everything downstream.
373+
374+
Detection is strictly declarative. CrewAI never guesses whether a string "looks
375+
like" an error, so a tool that legitimately returns text about an error is never
376+
misread as having failed. Failures are recorded when a tool returns a
377+
`ToolFailure`, when a tool raises, when an MCP server sets `isError`, when a
378+
tool's `max_usage_count` is spent, or when the agent calls a tool that does not exist.
379+
380+
### Choosing a Failure Policy
381+
382+
`tool_failure_policy` controls what happens next:
383+
384+
| Policy | Behavior |
385+
| :-- | :-- |
386+
| `ignore` | Nothing is recorded, emitted, or acted on. |
387+
| `warn` *(default)* | Records the failure, emits `ToolFailureDetectedEvent`, and continues. |
388+
| `raise` | Records and emits, then aborts with `ToolExecutionFailedError`. |
389+
390+
```python Code
391+
from crewai import Agent, Task
392+
from crewai.tools.tool_failure import ToolFailurePolicy
393+
394+
agent = Agent(
395+
role="Slack Messenger",
396+
goal="Post the report to Slack",
397+
backstory="...",
398+
tools=[SendSlackMessage()],
399+
tool_failure_policy=ToolFailurePolicy.WARN,
400+
)
401+
402+
# Tighten a single high-stakes task without changing the agent.
403+
task = Task(
404+
description="Post the final report to #engineering",
405+
expected_output="Confirmation the message was posted",
406+
agent=agent,
407+
tool_failure_policy=ToolFailurePolicy.RAISE,
408+
)
409+
```
410+
411+
The most specific setting wins: tool, then task, then agent, then crew, then the
412+
`warn` default.
413+
414+
### Inspecting Failures
415+
416+
Recorded failures are structured, so nothing downstream has to parse a string:
417+
418+
```python Code
419+
result = crew.kickoff()
420+
421+
if result.has_tool_failures:
422+
for record in result.tool_failures:
423+
print(record.tool_name) # "send_slack_message"
424+
print(record.failure.code) # "channel_not_found"
425+
print(record.failure.reason) # ToolFailureReason.TOOL_REPORTED
426+
print(record.summary())
427+
```
428+
429+
`tool_failures` is available on `TaskOutput`, `CrewOutput`, and
430+
`LiteAgentOutput`. A crew can finish successfully with a non-empty list — check
431+
it before treating `raw` as complete.
432+
433+
To react as failures happen, subscribe to the event:
434+
435+
```python Code
436+
from crewai.events import ToolFailureDetectedEvent
437+
from crewai.events.event_bus import crewai_event_bus
438+
439+
440+
@crewai_event_bus.on(ToolFailureDetectedEvent)
441+
def on_tool_failure(source, event):
442+
print(f"{event.tool_name} failed: {event.failure.message} ({event.policy})")
443+
```
444+
445+
The event is emitted before the `raise` policy aborts, so subscribers always
446+
observe the failure. `ToolUsageFinishedEvent` also carries a `failure` field, letting
447+
a trace UI mark the call as failed without correlating two events.
448+
337449
## Conclusion
338450

339451
Tools are pivotal in extending the capabilities of CrewAI agents, enabling them to undertake a broad spectrum of tasks and collaborate effectively.

lib/crewai-tools/src/crewai_tools/tools/crewai_platform_tools/crewai_platform_action_tool.py

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55
from typing import Any
66

77
from crewai.tools import BaseTool
8+
from crewai.tools.tool_failure import ToolFailure
89
from crewai.utilities.pydantic_schema_utils import create_model_from_schema
910
from pydantic import Field, create_model
1011
import requests
@@ -49,7 +50,7 @@ def __init__(
4950
self.action_name = action_name
5051
self.action_schema = action_schema
5152

52-
def _run(self, **kwargs: Any) -> str:
53+
def _run(self, **kwargs: Any) -> Any:
5354
try:
5455
cleaned_kwargs = {
5556
key: value for key, value in kwargs.items() if value is not None
@@ -85,9 +86,22 @@ def _run(self, **kwargs: Any) -> str:
8586
error_message = str(error_info)
8687
else:
8788
error_message = str(data)
88-
return f"API request failed: {error_message}"
89+
# The platform returns a non-2xx when the upstream app rejects
90+
# the action -- e.g. Slack answering channel_not_found. That is
91+
# the single most common way an agent "succeeds" at doing
92+
# nothing, so report it as a failure rather than as prose.
93+
return ToolFailure(
94+
message=f"API request failed: {error_message}",
95+
code=str(response.status_code),
96+
retryable=response.status_code >= 500,
97+
details={"action": self.action_name},
98+
)
8999

90100
return json.dumps(data, indent=2)
91101

92102
except Exception as e:
93-
return f"Error executing action {self.action_name}: {e!s}"
103+
return ToolFailure(
104+
message=f"Error executing action {self.action_name}: {e!s}",
105+
code=e.__class__.__name__,
106+
details={"action": self.action_name},
107+
)

lib/crewai/src/crewai/agent/core.py

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,7 @@
8686
from crewai.skills.models import INSTRUCTIONS, Skill as SkillModel
8787
from crewai.state.checkpoint_config import CheckpointConfig, apply_checkpoint
8888
from crewai.tools.agent_tools.agent_tools import AgentTools
89+
from crewai.tools.tool_failure import ToolExecutionFailedError
8990
from crewai.types.callback import SerializableCallable
9091
from crewai.types.usage_metrics import UsageMetrics
9192
from crewai.utilities.agent_utils import (
@@ -131,7 +132,10 @@
131132
from crewai.utilities.types import LLMMessage
132133

133134

134-
_passthrough_exceptions: tuple[type[Exception], ...] = ()
135+
# Exceptions that must not be swallowed into the max_retry_limit loop.
136+
# A tool_failure_policy="raise" abort is a deliberate stop, not a transient
137+
# error worth re-running the whole task for.
138+
_passthrough_exceptions: tuple[type[Exception], ...] = (ToolExecutionFailedError,)
135139

136140
_EXECUTOR_CLASS_MAP: dict[str, type] = {
137141
"CrewAgentExecutor": CrewAgentExecutor,
@@ -550,6 +554,8 @@ def _prepare_task_execution(
550554

551555
self._inject_date_to_task(task)
552556

557+
self.reset_tool_failures()
558+
553559
if self.tools_handler:
554560
self.tools_handler.last_used_tool = None
555561

lib/crewai/src/crewai/agents/agent_builder/base_agent.py

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,7 @@
4444
from crewai.skills.models import Skill
4545
from crewai.state.checkpoint_config import CheckpointConfig, _coerce_checkpoint
4646
from crewai.tools.base_tool import BaseTool, Tool
47+
from crewai.tools.tool_failure import ToolFailurePolicy, ToolFailureRecord
4748
from crewai.types.callback import SerializableCallable
4849
from crewai.utilities.config import process_config
4950
from crewai.utilities.i18n import I18N, get_i18n
@@ -264,6 +265,7 @@ class BaseAgent(BaseModel, ABC, metaclass=AgentMeta):
264265
_original_backstory: str | None = PrivateAttr(default=None)
265266
_token_process: TokenProcess = PrivateAttr(default_factory=TokenProcess)
266267
_kickoff_event_id: str | None = PrivateAttr(default=None)
268+
_tool_failures: list[ToolFailureRecord] = PrivateAttr(default_factory=list)
267269
id: UUID4 = Field(default_factory=uuid.uuid4, frozen=True)
268270
role: str = Field(description="Role of the agent")
269271
goal: str = Field(description="Objective of the agent")
@@ -298,6 +300,18 @@ class BaseAgent(BaseModel, ABC, metaclass=AgentMeta):
298300
max_iter: int = Field(
299301
default=25, description="Maximum iterations for an agent to execute a task"
300302
)
303+
tool_failure_policy: ToolFailurePolicy = Field(
304+
default=ToolFailurePolicy.WARN,
305+
description=(
306+
"How to react when a tool runs to completion but reports that it "
307+
"failed (an upstream API rejecting the request, an MCP server "
308+
"setting isError, a platform action returning an error payload). "
309+
"'ignore' restores pre-1.16 behavior and records nothing; 'warn' "
310+
"records the failure, emits ToolFailureDetectedEvent and keeps "
311+
"going; 'raise' additionally aborts with ToolExecutionFailedError. "
312+
"A Task or a tool may override this for a narrower scope."
313+
),
314+
)
301315
agent_executor: Annotated[
302316
SerializeAsAny[BaseAgentExecutor] | None,
303317
BeforeValidator(_validate_executor_ref),
@@ -652,6 +666,20 @@ def key(self) -> str:
652666
]
653667
return md5("|".join(source).encode(), usedforsecurity=False).hexdigest()
654668

669+
@property
670+
def last_tool_failures(self) -> list[ToolFailureRecord]:
671+
"""Tool failures recorded during the most recent execution.
672+
673+
Empty when nothing failed, or when ``tool_failure_policy`` is
674+
``ignore``. Reset at the start of each task execution, mirroring
675+
``last_messages``.
676+
"""
677+
return self._tool_failures
678+
679+
def reset_tool_failures(self) -> None:
680+
"""Clear recorded tool failures before a new execution begins."""
681+
self._tool_failures = []
682+
655683
@abstractmethod
656684
def execute_task(
657685
self,

lib/crewai/src/crewai/agents/crew_agent_executor.py

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,14 @@
4949
run_after_tool_call_hooks,
5050
run_before_tool_call_hooks,
5151
)
52+
from crewai.tools.tool_failure import (
53+
ToolExecutionFailedError,
54+
ToolFailure,
55+
ToolFailureReason,
56+
detect_tool_failure,
57+
failure_from_exception,
58+
handle_tool_failure,
59+
)
5260
from crewai.types.callback import SerializableCallable
5361
from crewai.utilities.agent_utils import (
5462
_llm_stop_words_applied,
@@ -431,6 +439,12 @@ def _invoke_loop_react(self) -> AgentFinish:
431439
self._invoke_step_callback(formatted_answer)
432440
self._append_message(formatted_answer.text)
433441

442+
except ToolExecutionFailedError:
443+
# tool_failure_policy="raise" asked for the run to stop; the
444+
# generic handler below would otherwise feed it back to the
445+
# LLM as a recoverable observation.
446+
raise
447+
434448
except OutputParserError as e:
435449
formatted_answer = handle_output_parser_exception( # type: ignore[assignment]
436450
e=e,
@@ -925,6 +939,7 @@ def _execute_single_native_tool_call(
925939
from_cache = False
926940
result: str = "Tool not found"
927941
raw_tool_result: Any = result
942+
tool_failure: ToolFailure | None = None
928943
input_str = json.dumps(args_dict) if args_dict else ""
929944
if self.tools_handler and self.tools_handler.cache and output_tool is not None:
930945
cached_result = self.tools_handler.cache.read(
@@ -933,6 +948,7 @@ def _execute_single_native_tool_call(
933948
if cached_result is not None:
934949
raw_tool_result = cached_result
935950
result = format_native_tool_output_for_agent(output_tool, cached_result)
951+
tool_failure = detect_tool_failure(cached_result)
936952
from_cache = True
937953

938954
agent_key = getattr(self.agent, "key", "unknown") if self.agent else "unknown"
@@ -967,6 +983,9 @@ def _execute_single_native_tool_call(
967983
elif max_usage_reached and original_tool:
968984
result = f"Tool '{func_name}' has reached its usage limit of {original_tool.max_usage_count} times and cannot be used anymore."
969985
raw_tool_result = result
986+
tool_failure = ToolFailure(
987+
message=result, reason=ToolFailureReason.USAGE_LIMIT
988+
)
970989
elif (
971990
not from_cache
972991
and func_name in available_functions
@@ -992,9 +1011,11 @@ def _execute_single_native_tool_call(
9921011
)
9931012

9941013
result = format_native_tool_output_for_agent(output_tool, raw_result)
1014+
tool_failure = detect_tool_failure(raw_result)
9951015
except Exception as e:
9961016
result = f"Error executing tool: {e}"
9971017
raw_tool_result = result
1018+
tool_failure = failure_from_exception(e)
9981019
if self.task:
9991020
self.task.increment_tools_errors()
10001021
crewai_event_bus.emit(
@@ -1036,9 +1057,23 @@ def _execute_single_native_tool_call(
10361057
agent_key=agent_key,
10371058
started_at=started_at,
10381059
finished_at=datetime.now(),
1060+
failure=tool_failure,
10391061
),
10401062
)
10411063

1064+
# After the hooks and the finished event, so subscribers see the full
1065+
# lifecycle even when the policy is about to abort the run.
1066+
if tool_failure is not None:
1067+
handle_tool_failure(
1068+
tool_failure,
1069+
tool_name=func_name,
1070+
tool_args=args_dict,
1071+
tool=structured_tool,
1072+
agent=self.agent,
1073+
task=self.task,
1074+
crew=self.crew,
1075+
)
1076+
10421077
return {
10431078
"call_id": call_id,
10441079
"func_name": func_name,
@@ -1246,6 +1281,12 @@ async def _ainvoke_loop_react(self) -> AgentFinish:
12461281
await self._ainvoke_step_callback(formatted_answer)
12471282
self._append_message(formatted_answer.text)
12481283

1284+
except ToolExecutionFailedError:
1285+
# tool_failure_policy="raise" asked for the run to stop; the
1286+
# generic handler below would otherwise feed it back to the
1287+
# LLM as a recoverable observation.
1288+
raise
1289+
12491290
except OutputParserError as e:
12501291
formatted_answer = handle_output_parser_exception( # type: ignore[assignment]
12511292
e=e,

lib/crewai/src/crewai/crews/crew_output.py

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77

88
from crewai.tasks.output_format import OutputFormat
99
from crewai.tasks.task_output import TaskOutput
10+
from crewai.tools.tool_failure import ToolFailureRecord
1011
from crewai.types.usage_metrics import UsageMetrics
1112

1213

@@ -31,6 +32,21 @@ class CrewOutput(BaseModel):
3132
default_factory=UsageMetrics,
3233
)
3334

35+
@property
36+
def tool_failures(self) -> list[ToolFailureRecord]:
37+
"""Every tool failure recorded across all tasks, in task order.
38+
39+
A crew can finish with a non-empty list: agents routinely narrate a
40+
failed step in prose and carry on, which used to make the run look
41+
entirely successful. Check this before treating ``raw`` as complete.
42+
"""
43+
return [failure for task in self.tasks_output for failure in task.tool_failures]
44+
45+
@property
46+
def has_tool_failures(self) -> bool:
47+
"""Whether any tool reported a failure during this crew run."""
48+
return any(task.tool_failures for task in self.tasks_output)
49+
3450
@property
3551
def usage_metrics(self) -> dict[str, Any]:
3652
"""Token usage as a plain dict.

0 commit comments

Comments
 (0)