📚 RAG Model Assessment / Document Q&A System

A modular, production-grade Retrieval-Augmented Generation (RAG) system built over Sean Carroll's physics textbook Spacetime and Geometry: An Introduction to General Relativity.

This repository features a clean, decoupled function pipeline in rag_pipeline.ipynb, engineered to be lifted into production microservices, FastAPI endpoints, or background workers with minimal refactoring.

---

📋 Table of Contents

• 🏗️ Pipeline Architecture
• ✨ Key Upgrades & Features
• 📊 Empirical Evaluation Results
• 🛠️ Models & Tech Stack
• 🚀 Quick Start & Usage
• 🔬 Core Pipeline API Functions
• ⚠️ Known Limitations
• 🔮 Future Roadmap

---

🏗️ Pipeline Architecture

                               ┌───────────────────────────┐
                               │      Carroll_SG.pdf       │
                               └─────────────┬─────────────┘
                                             │
                                  pypdf / PyMuPDF + OCR
                                             ▼
                               ┌───────────────────────────┐
                               │  Full Document Text Stream│
                               └─────────────┬─────────────┘
                                             │
                          RecursiveCharacterTextSplitter (chunk=500, overlap=50)
                                             ▼
                               ┌───────────────────────────┐
                               │   Text Chunks (3,208)     │
                               └──────┬─────────────────┬──┘
                                      │                 │
         SentenceTransformer          │                 │        rank_bm25
          (all-MiniLM-L6-v2)          │                 │       (BM25Okapi)
                      ▼               │                 │            ▼
         ┌─────────────────────────┐  │                 │  ┌───────────────────┐
         │ FAISS Vector Index      │◄─┘                 └─►│ BM25 Sparse Index │
         │ (IndexFlatL2 Dense)     │                       │ (Lexical Search)  │
         └────────────┬────────────┘                       └─────────┬─────────┘
                      │                                              │
                      └───────────────────┐      ┌───────────────────┐
                                          ▼      ▼
                               ┌───────────────────────────┐
                               │ Reciprocal Rank Fusion    │
                               │ (RRF Hybrid Retrieval)    │
                               └─────────────┬─────────────┘
                                             │
                                    Grounding Guard
                          (Cosine Similarity Threshold >= 0.40)
                                             │
                         ┌───────────────────┴───────────────────┐
                         ▼                                       ▼
                 [Pass Threshold]                         [Below Threshold]
                         │                                       │
           V2 Strict Grounding Prompt               "Insufficient information
           + Inline Chunk Citations                  available in document."
                         │
                         ▼
             Google Gemini LLM Engine
              (gemini-2.0-flash)
                         │
                         ▼
                 Grounded Answer

---

✨ Key Upgrades & Features

1. Hybrid Retrieval (RRF): Combines dense semantic vectors (FAISS L2) with sparse lexical keywords (BM25Okapi) using Reciprocal Rank Fusion (RRF): $$RRF_Score(d) = \frac{1}{k + Rank_{Dense}(d)} + \frac{1}{k + Rank_{BM25}(d)}$$
2. Grounding Guard: Evaluates cosine similarity between the query and top retrieved chunks. Rejects out-of-domain questions with "Insufficient information available in the document."
3. Prompt Versioning:
   • V1 (Plain): Basic context-conditioned query prompt.
   • V2 (Strict Grounding): Restricts answers strictly to provided context and enforces inline chunk citations ([Chunk 1], [Chunk 2]).
4. Custom Evaluation Framework: Evaluates 15 hand-curated domain questions across 4 quantitative metrics without expensive external LLM evaluators.
5. Latency & Cost Tracking: Log execution latency (s), prompt/completion tokens, and operational costs.

---

📊 Empirical Evaluation Results

Evaluated over 15 hand-curated physics Q&A pairs derived directly from Carroll's textbook:

Metric	Score	Description
Context Precision	0.840 (84.0%)	Proportion of retrieved chunks containing expected ground-truth keywords
Context Recall	0.731 (73.1%)	Fraction of ground-truth keywords captured across top-5 retrieved chunks
Faithfulness	0.831 (83.1%)	Overlap ratio between generated answer claims and retrieved context text
Answer Relevancy	0.328 (32.8%)	Cosine similarity between generated answer and ground-truth answer embeddings
Mean Latency	0.339s	Average end-to-end processing time per user query

---

🛠️ Models & Tech Stack

Component	Technology	Implementation Details
Text Ingestion	pypdf, fitz (PyMuPDF), pytesseract	Handles standard & scanned/image-based PDFs with OCR fallback
Text Chunking	langchain-text-splitters	RecursiveCharacterTextSplitter (chunk_size=500, chunk_overlap=50)
Vector Embeddings	sentence-transformers	all-MiniLM-L6-v2 (384 dimensions, 80MB footprint)
Dense Vector Store	faiss-cpu	IndexFlatL2 for fast Euclidean distance search
Sparse Index	rank-bm25	BM25Okapi over lower-cased tokenized corpus
LLM Engine	google-genai	gemini-2.0-flash with structured prompt grounding

---

🚀 Quick Start & Usage

1. Clone & Install Dependencies

git clone https://github.com/YMP7/RAG_Model_Assesment.git
cd RAG_Model_Assesment
pip install -r requirements.txt

2. Configure Environment

Create a .env file in the root directory:

GOOGLE_API_KEY=your_gemini_api_key_here

3. Launch Notebook

jupyter notebook rag_pipeline.ipynb

---

🔬 Core Pipeline API Functions

Every pipeline stage is implemented as a standalone function:

# 1. Document Ingestion
load_pdf(path: str, extracted_txt_path: str = None) -> str

# 2. Text Chunking
chunk_text(text: str, chunk_size: int = 500, chunk_overlap: int = 50) -> list[str]

# 3. Vector & Sparse Indexing
build_faiss_index(embeddings: np.ndarray) -> faiss.IndexFlatL2
build_bm25_index(chunks: list[str]) -> BM25Okapi

# 4. Hybrid Retrieval
retrieve_hybrid(query: str, faiss_index, bm25_index, chunks, model, top_k: int = 5) -> list[tuple[str, float]]

# 5. Guarded QA Pipeline
answer_query_guarded(query: str, index, bm25_index, chunks, embed_model, client, threshold: float = 0.40) -> tuple[str, float, dict]

---

⚠️ Known Limitations

1. OCR Artifacts on LaTeX Equations: Inline mathematical symbols (e.g., $\Gamma^\lambda_{\mu\nu}$, $\nabla_\mu T^{\mu\nu}$) are extracted as raw text characters by OCR, which can lower term-matching precision on purely mathematical queries.
2. Non-Text Diagrams: Figures, commutative diagrams, and tensor flowcharts are converted to line text; visual diagram information is omitted.
3. Single Document Scope: Configured for single document QA. Multi-document support requires document ID metadata indexing.

---

🔮 Future Roadmap

• FastAPI Microservice: Wrap answer_query_guarded into a POST /v1/query REST endpoint.
• Cross-Encoder Reranker: Add a ms-marco-MiniLM-L-6-v2 cross-encoder stage after RRF.
• Multimodal Ingestion: Utilize Gemini 2.0 Flash Vision to process mathematical figures and tensor diagrams directly.

---

Developed for the RAG Model Assessment / Document Q&A System.