🔍 DocLens

Upload a PDF, ask it questions, and get answers grounded only in that document — with the exact page, excerpt, and supporting sentence highlighted. Never a guess from outside knowledge.

Built as a second portfolio project to demonstrate real Retrieval-Augmented Generation (RAG) — not a chatbot with a PDF attached, but an actual retrieval pipeline: extraction, chunking, embeddings, vector search, and grounded generation, with every number the user sees calculated from real evidence rather than trusted from an AI's self-report.

---

The Problem

Reading a long document to find one specific answer is slow. Asking a general AI chatbot instead is fast — but risky, since it can confidently invent something that sounds right but isn't actually in the document. DocLens solves this by forcing the model to answer only from retrieved, cited excerpts of the uploaded file, and by proving it: every answer shows the exact page, excerpt, and sentence it came from.

How It Works

User
  │
Upload PDF
  │
Extract Text (page by page, via PyMuPDF)
  │
Chunk Document (overlapping word-based chunks, never crossing a page boundary)
  │
Generate Embeddings (Gemini Embedding API — gemini-embedding-001)
  │
Store in FAISS (L2-normalized vectors, inner product = cosine similarity)
  │
User Question
  │
Question Embedding (asymmetric: RETRIEVAL_QUERY vs. RETRIEVAL_DOCUMENT)
  │
Semantic Search (top-k most similar chunks)
  │
Calculate Confidence (from the real similarity score — not asked of the AI)
  │
Send Retrieved Chunks to Gemini (generation is constrained to only this context)
  │
Grounded Answer + Exact Supporting Sentence (copied verbatim, for highlighting)
  │
Display: Answer + Citation (📄 file, page, excerpt with sentence highlighted) + Confidence

A Real Engineering Decision, Not Just a Prompt

Confidence isn't asked of the AI — it's calculated from the actual cosine similarity score of the retrieved evidence:

Similarity Score	Confidence
Above 0.90	High
0.75 – 0.90	Medium
Below 0.75	Low

This is the same principle used in this developer's first project (Scam Smell Test), where the risk score is summed from evidence points rather than trusted as a raw AI-stated number: numbers the user sees should be derived from real signal, not asked of the model on faith.

The same principle shows up again in the document summarizer: recommendations are tagged High/Medium/Low priority, not given a fabricated 5-star "impact score" — a precise-looking number with no real measurement behind it is worse than an honest, coarser signal.

Features

• 📄 PDF upload and automatic indexing — extraction, chunking, embedding, and vector storage, fully automatic
• 📋 Whole-document summarization — a genuinely separate pipeline from retrieval-QA: a single direct call for documents that fit in one context window, or real map-reduce (summarize batches of pages, then combine) for longer ones. Not the same code path as question-answering, because top-k retrieval is the wrong tool for "summarize everything"
• 🎯 Grounded answers only — the model is constrained to the retrieved excerpts and explicitly told to say so honestly if the document doesn't contain the answer, rather than guessing
• 🖍️ Exact sentence highlighting — not just "here's the paragraph," but the precise sentence the answer is based on, verified as an exact substring before display
• 📊 Real, calculated confidence — derived from retrieval similarity scores, not AI self-report
• 💸 $0 to run — Gemini's embedding and generation models both have genuine free tiers, no credit card required

Known Limitations

Stated openly, not hidden:

• PDF only (no DOCX, TXT, etc. in v1)
• Text-based PDFs only — scanned/image PDFs need OCR, which isn't included in v1
• Single document at a time — no cross-document comparison yet
• No image or table extraction — body text only
• Primarily tested/tuned for English
• Best at direct factual lookup, not open-ended reframing — the strict grounding rule that prevents hallucination on questions like "which tool generated this?" also makes the system decline requests like "explain this for a business owner," even when the underlying facts are present in the retrieved text. This is a deliberate trade-off for v1, not a bug: loosening it risks quietly reopening the door to unsupported claims. A dedicated synthesis mode is a natural v2 feature, not a quick prompt tweak.
• Citation precision varies by document style — works best on prose; dense tables/reports with no real sentence boundaries can produce a wider "supporting sentence" span than a clean prose document would

Tech Stack

Layer	Technology
Frontend	Vanilla HTML / CSS / JavaScript
Backend	Python, FastAPI
PDF Parsing	PyMuPDF
Embeddings	Google Gemini (gemini-embedding-001)
Vector Search	FAISS (IndexFlatIP, local, in-memory)
Generation	Google Gemini (gemini-3.5-flash)
Structured Output	Gemini response_schema

Getting Started

1. Clone this repo
2. Get a free Gemini API key at aistudio.google.com/app/apikey (or reuse an existing one)
3. cd backend && cp .env.example .env and paste your key into .env
4. pip install -r requirements.txt
5. uvicorn main:app --reload --port 8000
6. Open frontend/index.html in your browser

Project Structure

doclens/
├── README.md
├── PROJECT_PLAN.md
├── backend/
│   ├── main.py           # FastAPI app: /api/upload, /api/ask, /api/summarize
│   ├── rag.py             # Chunking, embedding, FAISS retrieval, and summarization logic
│   ├── prompts.py         # Grounded-answer and summarization system prompts
│   ├── requirements.txt
│   └── .env.example
└── frontend/
    └── index.html

Future Improvements

• Multiple documents + cross-document comparison
• Quiz / flashcard generation from uploaded material
• Conversation memory (follow-up questions, pronoun resolution)
• OCR support for scanned PDFs
• Search history, drag-and-drop upload polish

License

MIT — free to use, modify, and learn from.