🏦 Banking RAG Chatbot

A production-ready Retrieval-Augmented Generation (RAG) chatbot for banking products, built with FastAPI, LangChain, ChromaDB, and Groq. Features multilingual support (English + Burmese), automatic query routing, real-time evaluation, observability monitoring, conversation memory, and a full analytics dashboard.

---

📋 Table of Contents

• Architecture
• RAG Pipeline
• Evaluation Pipeline
• Observability Pipeline
• Query Routing
• Multilingual Support
• Metrics Explained
• Tech Stack
• Project Structure
• Getting Started
• API Endpoints
• Future Roadmap

---

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        User Interface                           │
│  Chat UI  │  Analytics Dashboard  │  Observability  │  Admin   │
└─────────────────────┬───────────────────────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────────────────────┐
│                     FastAPI Backend                             │
│   /chat   │   /ingest   │   /evaluate   │   /feedback          │
│   /sessions   │   /dashboard   │   /observability              │
└─────────────────────┬───────────────────────────────────────────┘
                      │
        ┌─────────────┼─────────────┐
        │             │             │
┌───────▼──────┐ ┌────▼────┐ ┌────▼──────────┐
│  LangChain   │ │ChromaDB │ │  Groq LLMs    │
│  RAG Chain   │ │Vectors  │ │  Llama 3.3    │
│  + Routing   │ │741 chunks│ │  Qwen 3 32B  │
└──────────────┘ └─────────┘ └───────────────┘
        │
┌───────▼──────────────────────────────────────┐
│            Logging & Observability            │
│  eval_log  │  feedback_log  │  obs_log       │
└──────────────────────────────────────────────┘

---

RAG Pipeline

User Query
    │
    ▼
Language Detection
(Unicode range check → English or Burmese)
    │
    ▼
Query Router
    ├── Simple / Factual ──────────► Llama 3.3 70B (fast retrieval)
    └── Complex / Comparison ──────► Qwen 3 32B (deep reasoning)
    │
    ▼
ChromaDB Similarity Search
(HuggingFace all-MiniLM-L6-v2 embeddings)
    │
    ▼
Top-K Chunks Retrieved (k=15)
+ Similarity Scores
+ Source Metadata
    │
    ▼
Prompt Builder
(System prompt + language instruction + context + chat history)
    │
    ▼
LLM Response Generation (streaming)
    │
    ├──► Evaluation Pipeline (optional, per-query)
    ├──► Observability Logger (always)
    └──► Session Memory (always)
    │
    ▼
Final Response + Source Citations

---

Evaluation Pipeline

Query + Retrieved Context + Generated Answer
    │
    ▼
┌───────────────────────────────────────────┐
│           LLM Judge (Llama 4 Scout)       │
│  Bilingual-aware — handles Burmese context│
└───────────────────────────────────────────┘
    │
    ├──► Context Relevance
    │    Are retrieved chunks relevant to the question?
    │    Score: 0.0 – 1.0
    │
    ├──► Faithfulness
    │    Is the answer grounded in the context?
    │    Score: 0.0 – 1.0
    │
    ├──► Answer Relevance
    │    Does the answer address the question?
    │    Score: 0.0 – 1.0
    │
    ├──► Hallucination Detection
    │    LLM Judge + Rule-based word overlap
    │    Score: 0.0 (hallucinated) – 1.0 (clean)
    │
    └──► Retrieval Quality
         Precision, Recall, MRR
         Rule-based keyword overlap scoring
    │
    ▼
Overall Score (average of LLM-based metrics)
    │
    ▼
Logged to eval_log.csv + eval_log.json

---

Observability Pipeline

Every Inference Request
    │
    ▼
┌──────────────────────────────────────────────┐
│              Latency Monitor                 │
│  retrieval_latency_ms                        │
│  llm_latency_ms                              │
│  total_latency_ms                            │
│  P50 / P95 / P99 percentiles                 │
└──────────────────────────────────────────────┘
    │
    ▼
┌──────────────────────────────────────────────┐
│              Token Tracker                   │
│  estimated_prompt_tokens                     │
│  estimated_completion_tokens                 │
│  estimated_total_tokens                      │
│  cumulative total across all queries         │
└──────────────────────────────────────────────┘
    │
    ▼
┌──────────────────────────────────────────────┐
│           Retrieval Diagnostics              │
│  top_score, avg_score, min_score             │
│  chunks_retrieved, chunks_used               │
│  top_chunk_source, top_chunk_section         │
│  full retrieved_docs list with scores        │
└──────────────────────────────────────────────┘
    │
    ▼
┌──────────────────────────────────────────────┐
│             Failure Analysis                 │
│  retrieval_success (score > 0.2)             │
│  grounded_retrieval (answer ∩ context)       │
│  failed_retrieval                            │
│  failure_reason:                             │
│    - unsupported_query                       │
│    - low_confidence                          │
│    - empty_retrieval                         │
└──────────────────────────────────────────────┘
    │
    ▼
Logged to observability_log.csv + observability_log.json
    │
    ▼
Observability Dashboard
(Latency trends, token usage, score distribution,
 failure table, retrieval diagnostics per query)

---

Query Routing

The router classifies every question before sending it to an LLM. Simple factual questions use the fast retrieval model; complex multi-step questions use the reasoning model.

Routing logic:

REASONING_KEYWORDS = [
    # Comparison
    "compare", "difference", "better", "best", "vs", "versus", "which",
    # Multi-step
    "calculate", "how much", "total", "eligibility", "qualify",
    # Conditional
    "if i", "what happens if", "can i", "should i",
    # Complex
    "explain", "why", "recommend", "advise", "pros and cons"
]

Question Type	Example	Model
Factual lookup	"What is the interest rate for term loan?"	⚡ Llama 3.3 70B
Comparison	"Which loan is better for 2 years?"	🧠 Qwen 3 32B
Eligibility	"Can I apply for a credit card?"	🧠 Qwen 3 32B
Definition	"What is a fixed deposit account?"	⚡ Llama 3.3 70B
Recommendation	"Which account should I open?"	🧠 Qwen 3 32B

---

Multilingual Support

Language detection uses Unicode character range analysis — no external library needed.

User sends a question
    │
    ▼
Count Burmese Unicode characters (U+1000 – U+109F)
    │
    ├── ratio > 20% ──► Language = Burmese (မြန်မာဘာသာ)
    └── ratio ≤ 20% ──► Language = English
    │
    ▼
Language injected into system prompt:
"You MUST respond in {language}"
    │
    ▼
LLM answers in the detected language
Source citations remain in original Burmese

Supported languages: English, Burmese (မြန်မာဘာသာ)

Example:

Input	Detected	Output
"What is the interest rate?"	🇬🇧 English	English answer
"အတိုးနှုန်း ဘယ်လောက်လဲ?"	🇲🇲 Burmese	Burmese answer
"CB Pay ကို ဘယ်လို register လုပ်ရမလဲ?"	🇲🇲 Burmese	Burmese answer

---

Metrics Explained

Evaluation Metrics

Metric	Description	Range	Good Score
Context Relevance	Are retrieved chunks relevant to the question?	0 – 1	> 0.7
Faithfulness	Is the answer supported by the context?	0 – 1	> 0.8
Answer Relevance	Does the answer address the question asked?	0 – 1	> 0.8
Hallucination Score	Combined LLM judge + rule-based overlap check	0 – 1	> 0.7
Overall Score	Average of the four LLM-based metrics above	0 – 1	> 0.7

Retrieval Metrics

Metric	Description
Precision	Fraction of retrieved chunks with keyword overlap > 20%
Coverage	Average overlap score across all retrieved chunks
MRR	Mean Reciprocal Rank — how high the first relevant chunk ranks
Top Score	Highest similarity score among retrieved chunks (0 = no match, 1 = perfect)

Observability Metrics

Metric	Description
Retrieval Latency	Time to embed query + search ChromaDB
LLM Latency	Time for LLM to generate the full response
Total Latency	End-to-end response time
P95 Latency	95th percentile — worst-case latency for 95% of queries
Grounded Rate	Fraction of answers where content overlaps with retrieved context
Failed Retrieval Rate	Fraction of queries where retrieval returned low-confidence or unsupported results
Estimated Tokens	Prompt + completion token count (1 token ≈ 4 characters)

---

Document Ingestion

The ingestion pipeline handles 3 Q&A document formats automatically:

Format 1 — Q:/A: style

Q: Term Loan ဆိုတာဘာလဲ?
A: Term Loan သည် ...

Format 2 — Structured metadata style

Category: Loans
Product: Term Loan
Topic: Overview
Question:
What is a Term Loan?
Alternative Questions:
• Term loan ဆိုတာဘာလဲ
Answer:
A Term Loan is...

Format 3 — Plain FAQ

What is Personal Internet Banking?
CB Bank's Personal Internet Banking is...

Alternative questions are indexed as separate searchable chunks, each containing the full answer. This dramatically improves retrieval for varied phrasings.

---

Tech Stack

Component	Technology
API Framework	FastAPI + Uvicorn
RAG Orchestration	LangChain
Vector Store	ChromaDB (local)
Embeddings	HuggingFace all-MiniLM-L6-v2 (free, local)
LLM — Retrieval	Groq Llama 3.3 70B
LLM — Reasoning	Groq Qwen 3 32B
LLM — Evaluation	Groq Llama 4 Scout 17B
Document Parsing	PyMuPDF (PDF), python-docx (DOCX)
Frontend	Vanilla HTML/CSS/JS + Chart.js
Session Storage	JSON files
Log Storage	CSV + JSON

---

Project Structure

banking-rag-chatbot/
├── app/
│   ├── config.py             # Settings, model names, admin password
│   ├── ingest.py             # Document parsing (3 formats), ChromaDB ingestion
│   ├── chain.py              # RAG chain, query routing, multilingual detection
│   ├── evaluator.py          # 5 evaluation metrics, LLM judge
│   ├── logger.py             # Evaluation log (CSV + JSON)
│   ├── feedback.py           # User feedback log (CSV + JSON)
│   ├── session_manager.py    # Conversation memory, context window management
│   ├── observability.py      # Latency, token, retrieval diagnostics logging
│   └── main.py               # FastAPI app, all endpoints
│
├── frontend/
│   ├── index.html            # Streaming chat UI
│   ├── dashboard.html        # Analytics dashboard
│   ├── observability.html    # Observability monitoring page
│   └── admin.html            # Password-protected admin panel
│
├── docs/                     # Product knowledge base
│   ├── accounts/             # Account product documents
│   ├── cards/                # Card product documents
│   ├── loans/                # Loan product documents
│   ├── mobile_banking/       # Mobile banking documents
│   └── ibanking/             # Internet banking documents
│
├── sessions/                 # Conversation session JSON files
│
├── logs/
│   ├── eval_log.csv          # Evaluation scores per query
│   ├── eval_log.json
│   ├── feedback_log.csv      # User 👍 👎 feedback
│   ├── feedback_log.json
│   ├── observability_log.csv # Latency, tokens, retrieval diagnostics
│   └── observability_log.json
│
├── .env                      # API keys (not committed)
├── .gitignore
└── requirements.txt

---

Getting Started

Prerequisites

• Python 3.10+
• Free Groq API key

Installation

git clone https://github.com/yourusername/banking-rag-chatbot
cd banking-rag-chatbot

python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate

pip install -r requirements.txt

Configuration

Create .env:

GROQ_API_KEY=your_groq_api_key_here
ADMIN_PASSWORD=your_admin_password

Add Documents

Place your PDF or DOCX files in the appropriate folder:

docs/accounts/    ← account product documents
docs/cards/       ← card product documents
docs/loans/       ← loan product documents
docs/mobile_banking/
docs/ibanking/

Ingest Documents

python -c "from app.ingest import ingest_documents; print(ingest_documents())"

Run

uvicorn app.main:app --reload --port 8000

Access

URL	Description
http://localhost:8000/chat-ui	Chat interface
http://localhost:8000/dashboard	Analytics dashboard
http://localhost:8000/observability	Observability monitoring
http://localhost:8000/admin	Admin panel (password protected)
http://localhost:8000/docs	Auto-generated API docs

---

API Endpoints

Method	Endpoint	Description
POST	/chat	Streaming chat with session memory
POST	/ingest	Upload and index a document
POST	/evaluate	Manual evaluation of a Q&A pair
POST	/feedback	Submit 👍 👎 feedback
GET	/sessions	List all sessions
POST	/sessions/new	Create a new session
GET	/sessions/{id}	Get session with full history
DELETE	/sessions/{id}	Delete a session
GET	/logs/summary	Evaluation summary statistics
GET	/logs/all	All evaluation records
GET	/feedback/summary	Feedback summary statistics
GET	/feedback/all	All feedback records
GET	/observability/summary	Observability summary
GET	/observability/all	All observation records
GET	/admin/files	List all ingested documents
DELETE	/admin/files/{cat}/{file}	Delete a document
POST	/admin/reingest	Re-ingest all documents
GET	/admin/stats	System-wide admin statistics
GET	/health	Health check

---

Future Roadmap

Short Term

• Convert scanned PDFs to structured DOCX for better extraction
• Add batch evaluation from CSV upload in admin panel
• Auto-generate overview documents per product category
• Add confidence score display in chat UI

Medium Term

• Deploy to cloud (Railway / Render / AWS)
• Add Redis for session storage (scale beyond single server)
• Implement re-ranking layer (Cohere or cross-encoder)
• Add user authentication and multi-tenant support
• Webhook notifications for low evaluation scores

Long Term

• Fine-tune embedding model on domain-specific Burmese banking text
• Agentic RAG — multi-hop reasoning across documents
• Voice input support for Burmese
• Integrate with live banking API for real-time product data
• A/B testing framework for prompt and model experiments