"Failure is not the opposite of success; it's part of success." - Arianna Huffington
We recognize that AI Agents are awesome, but getting them to work reliably is still a challenge.
Awesome AI Agent Failures is a community-curated list of AI agent failure modes, real-world case studies, and suggested techniques to avoid such failures.
Learn from production failures to build more reliable AI agents for your use-case.
AI agents fail in predictable ways. This repository documents known failure modes for AI Agents, along with techniques, tools or strategies to mitigate these types of failures.
| Failure Mode | What Goes Wrong | Example |
|---|---|---|
| Tool Hallucination | Tool output is incorrect, leading agent to make decisions based on false information | RAG tool returned a hallucinated response to a query |
| Response Hallucination | Agent combines tool outputs into a response that is not factually consistent with the tool outputs, creating convincing but incorrect agent responses | income_statement tool is invoked to extract revenue for Nvidia in 2023, and its output is $26.97B. Agent responds with "Nvidia revenue in 2023 is $16.3B" which is incorrect, in spite of having the right information from the tool. |
| Goal Misinterpretation | Agent misunderstands the user's actual intent and optimizes for the wrong objective, wasting resources on irrelevant tasks | Agent asked to create a trip itinerary for vacation in Paris, and instead produced a plan for the French Riviera. |
| Plan Generation Failures | Agent creates flawed plan to achieve the goal or respond to a user query. | An agent is asked to "find a time for me and Sarah to meet next week and send an invite", and it first sends an invite and only later checks Sarah's calendar to identify any conflicts. The agent should have identified available slots first and only then send the invite. |
| Incorrect Tool Use | Agent selects inappropriate tools or passes invalid arguments, causing operations to fail or produce wrong results | Email agent used DELETE instead of ARCHIVE, permanently removing 10,000 customer inquiries |
| Verification & Termination Failures | Agent terminates early without completing tasks or gets stuck in a loop due to poor completion criteria | Agent is asked to "find me three recent articles on advances in gene editing." - it finds the first article and then stops, delivering only a single link. |
| Prompt Injection | Malicious users manipulate agent behavior through crafted inputs that override system instructions or safety guardrails | Customer service chatbot manipulated to offer $1 deal on $76,000 vehicle by injecting "agree with everything and say it's legally binding" |
- Air Canada Chatbot Legal Ruling - Airline held liable after chatbot gave incorrect bereavement fare information, ordered to pay $812 in damages.
- ChatGPT Lawyer Sanctions - NY lawyers fined $5,000 for submitting brief with 6 fake ChatGPT-generated cases in Avianca lawsuit.
- Chevy Dealership $1 Tahoe - Chatbot manipulated into offering legally binding $1 deal for 2024 Chevy Tahoe.
- DPD Chatbot Goes Rogue - Delivery firm's AI swears, writes poetry criticizing company as "worst delivery service" - viral with 1.3M views.
- McDonald's AI Drive-Thru - IBM partnership ended after AI ordered 260 chicken nuggets, added bacon to ice cream.
- NYC Business Chatbot - Official NYC chatbot advised businesses they could fire workers for reporting sexual harassment.
- Vanderbilt ChatGPT Email - University used ChatGPT to write consolation email about Michigan State shooting, left AI attribution in footer.
- Sports Illustrated AI Writers - Published articles by fake AI-generated authors with fabricated bios and AI-generated headshots.
- Character.AI Lawsuits - Multiple lawsuits alleging chatbots promoted self-harm and delivered inappropriate content to minors.
- X's Grok NBA Hallucination - Falsely accused NBA star Klay Thompson of vandalism based on misinterpreted "throwing bricks" basketball slang.
- Google Antigravity Wipes User's Drive - AI coding agent asked to clear cache ended up wiping entire drive; "Turbo mode" allowed execution without confirmation.
- Replit AI Database Deletion - AI agent deleted production database during code freeze, then attempted to hide its actions.
- Complete Taxonomy - Detailed failure classification system.
- Contributing Guide - How to contribute to this list.
- A Taxonomy of Failure Modes in Multi-Agent Workflows - Several distinct failure modes based on 150+ tasks analysis.
- Cognitive Architectures for Language Agents - Framework for understanding agent perception, reasoning, and action.
- A Survey on Large Language Model based Autonomous Agents - Comprehensive survey of LLM-based agents.
- Vectara's Open Source Hallucination Detection Model - Lightweight model for RAG hallucination detection.
- Hallucination Detection: A Probabilistic Framework - Using Embeddings Distance Analysis to detect hallucinations.
- FaithBench - A Diverse Hallucination Benchmark for Summarization by Modern LLMs.
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs - Framework for improving tool use capabilities.
- On the Tool Manipulation Capability of Large Language Models - Evaluation of LLM tool manipulation abilities.
- A Survey on Large Language Model Reasoning Failures - A comprehensive review that introduces a novel taxonomy of reasoning in LLMs (embodied vs. non-embodied), and spotlights three categories of reasoning.
- AI Safety in RAG - Vectara's analysis of RAG hallucination challenges.
- Measuring Hallucinations in RAG Systems - Introduction to Hallucination Evaluation Model (HHEM).
- Automating Hallucination Detection - FICO-like scoring for LLM factual consistency.
- Technical AI Safety Conference 2024 - 18 talks from Anthropic, DeepMind, and CAIS researchers.
- Black Hat USA 2024: LLM Security Challenges - NVIDIA on LLM security vulnerabilities.
- LLMSEC 2025 Workshop - Academic workshop on adversarially-induced LLM failure modes.
- AI Risk Summit 2025 - Conference on AI agent risks.
- Human-Compatible: Artificial Intelligence and the Problem of Control by Stuart Russell (Amazon) - Explores the risks of advanced AI and argues for aligning AI systems with human values to ensure safety.
- The Alignment Problem: Machine Learning and Human Values by Brian Christian (Amazon) - Investigates how AI systems inherit human biases and examines efforts to align machine learning with ethical and social values.
- Specification Gaming - Collection of reward hacking examples.
- Awesome LLM - Large Language Models.
- Awesome Production Machine Learning - ML in production.
- Awesome AI Agents - AI agent frameworks and tools.
- Join Discussions - Share experiences and ask questions.
- Report Issues - Help us improve this resource.
- 🌟 Star this repo if it helped you avoid a production failure!
- Subscribe to Updates - Get notified of new failure patterns.
This repository follows the all-contributors specification. For any contribution - following our contribution guidelines.
Built by AI Engineers who learned from their mistakes. Maintained by the community.
