[Feat]: Add a RAG failure mode checklist doc (symptoms to minimal fixes)

[onestardao]

Question Validation

• I have searched both the documentation and discord for an answer.

Question

Hi maintainers, thanks for the project.

Problem Description
In production RAG apps, most failures are not “model is dumb”, it is a small set of repeatable failure modes:
retrieval hallucination, wrong chunk selected, index fragmentation, bootstrap ordering races, config drift, etc.
Users often cannot name the failure mode, so they patch randomly and lose time.

Desired Solution you'd like
Add a small doc page: “RAG failure mode checklist”.
Structure:

1. common symptoms users can observe
2. what to inspect first (retrieval outputs, chunking, embeddings, store health, tracing)
3. minimal structural fixes, not generic advice
4. a simple taxonomy mapping so users can quickly classify before changing infra

I can draft a first version as a docs PR if the direction is accepted.

Alternatives Considered
Rely on scattered troubleshooting notes and GitHub issues.
This works but it is not systematic, and new users repeat the same mistakes.

Additional Context
I have a compact 16 mode taxonomy (No.1 to No.16) that I can adapt into a neutral checklist format
with minimal references, so it stays useful even without adopting any external tooling.
If you prefer, we can keep it fully project native and only use it as an internal naming scheme.

https://github.com/onestardao/WFGY/blob/main/ProblemMap/README.md

[dgenio]

I’m happy to draft a docs PR for this.

Proposed doc shape (aim: fast triage, minimal “hand-wavy” advice):

• A short “first 5 checks” flow (what to print/log first: retrieved nodes, scores, chunk boundaries, embedding model, prompt/context size).
• A table: Symptom → Likely failure mode → What to inspect → Minimal structural fixes.
• Keep it project-native (no dependency on external tooling); links only to existing LlamaIndex docs/pages.

Two questions to align before I write:

1. Where do you want this to live? (e.g., “Docs → RAG → Troubleshooting” vs a general “Troubleshooting” section)
2. Do you prefer a compact taxonomy (10–16 named failure modes) or fewer “buckets” with more detail per bucket?

If you’re OK with that outline, I’ll open a PR with the first version and we can tune the categories/examples.

[ManasVardhan]

I'm working on this — adding a RAG Failure Mode Checklist documentation page covering common failure modes (retrieval hallucination, wrong chunk selection, index fragmentation, config drift, embedding mismatch, context window overflow, etc.) with symptoms and fixes for each. PR incoming shortly.

[onestardao]

thanks, this outline looks great.
for placement, i think Docs > RAG > Troubleshooting (or a troubleshooting section with a RAG page) makes the most sense.
for taxonomy, i prefer a small number of buckets for the doc, and we can keep any finer labels as optional internal naming, not required for users. happy to review the first PR draft when it’s up.

if helpful, i can draft a small initial taxonomy table or add a minimal “symptom → what to inspect” mapping as a follow-up PR once the first version is up. happy to collaborate on tuning the buckets/examples.