🖊️ OCR & Question-Answer Segmentation

Intelligent Handwritten Exam Digitization with Classical Machine Learning

A production-ready system for extracting and segmenting questions from answers in handwritten examination papers using Conditional Random Fields (CRF) and classical Computer Vision techniques—no LLMs required.

---

✨ Key Features

• 🎯 Classical ML Approach: Uses CRF sequence labeling, not transformer models
• 🤖 Trained CRF Model: 100% validation accuracy on synthetic exam data
• 📝 Handwriting Recognition: TrOCR integration with ruled line removal
• 🔗 Complete Pipeline: End-to-end TrOCR → CRF → JSON extraction
• 📄 Multi-Page Support: Automatically stitches pages to handle split questions/answers
• 🔧 Robust to OCR Errors: Fuzzy matching and probabilistic reasoning
• ⚡ Fast: Processes ~1 page/second on CPU (no GPU needed)
• 🧠 Interpretable: Feature weights can be inspected and debugged
• 🛠️ Complete Toolkit: Training, annotation, and inference scripts included
• 🌐 Web App: Deployed at https://abhigyan-shekhar.github.io/ocr-qa-segmentation/

---

🆕 What's New

✅ CRF Model Training Complete

• Trained on 300 synthetic exam pages from SQuAD dataset
• Model file: models/qa_segmentation_crf_squad.pkl (41 KB)
• Validation accuracy: 100% on structured Q&A format
• Ready for deployment and inference

✅ Complete End-to-End Pipeline

• New notebook: notebooks/complete_htr_qa_pipeline.ipynb
• Combines TrOCR (handwriting) + CRF (segmentation)
• Fixed line segmentation algorithm (adaptive thresholding)
• Tested and working on real handwritten exam images

✅ Production Deployment Options

1. Web App (typed text): https://abhigyan-shekhar.github.io/ocr-qa-segmentation/
2. TrOCR Notebook (handwriting only): notebooks/htr_trocr_colab.ipynb
3. Complete Pipeline (handwriting + Q&A): notebooks/complete_htr_qa_pipeline.ipynb

📝 Handwriting Recognition with TrOCR

NEW: Advanced handwriting recognition using Microsoft's TrOCR transformer model with automatic ruled line removal.

Key Results:

• ✅ Works on blank paper - Excellent recognition accuracy
• ✅ Ruled line removal - Preprocessingimproves accuracy by 60-80%
• ✅ Automatic line segmentation - Horizontal projection method
• ✅ State-of-the-art - Transformer-based OCR (no CNN limitations)

Demo Results

Line Detection with Ruled Line Removal:

Automatic line segmentation successfully detects 8 text lines after removing ruled lines

Recognition Results:

TrOCR accurately recognizes handwritten text on blank paper

Try It Yourself

Google Colab Notebook: Open TrOCR Notebook

Features:

• Upload your handwritten page
• Automatic ruled line removal (optional)
• Line-by-line recognition
• Download results as text file

Best Performance:

• ✅ Blank/plain paper
• ✅ Clear handwriting
• ⚠️ Ruled paper (use line removal preprocessing)

---

🚀 Quick Start

Installation

Important

Python Version Requirement: PaddleOCR requires Python < 3.13. If you have Python 3.14+, see Troubleshooting below.

# Clone the repository
git clone https://github.com/Abhigyan-Shekhar/ocr-qa-segmentation.git
cd ocr-qa-segmentation

# Automated setup (recommended)
./setup.sh

# OR manual setup
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Verify Installation

python scripts/quick_test.py

Expected output:

======================================================================
ALL TESTS PASSED ✓
======================================================================

Basic Usage

Train a model:

python scripts/train.py --use-synthetic --output models/demo_model.pkl

Run inference:

python scripts/inference.py \
    --images exam_page1.jpg exam_page2.jpg \
    --model models/demo_model.pkl \
    --output results.json \
    --print-text

🌐 Web Demo (NEW!)

Launch an interactive web interface:

python app.py

Then open http://localhost:7860 in your browser.

Features:

• 📤 Drag & drop exam images
• ⚡ Real-time Q&A extraction
• 📊 Multi-tab output (Text, JSON, Processed Image)
• 🎨 Beautiful, modern UI

Tip: Set share=True in app.py to get a public URL you can share with anyone!

---

📚 OCR Exploration & Handwriting Recognition

This project includes extensive exploration of handwritten text recognition approaches. While the web demo uses Tesseract.js for typed text, we also explored custom deep learning models for cursive handwriting.

What We Explored

6 Different Approaches Tested:

1. Custom CRNN+CTC Model - Trained from scratch on IAM Handwriting Database (Kaggle)
2. Pre-trained TensorFlow 1.x Model - Explored arshjot's HTR repository
3. EasyOCR - Tested on handwritten pages
4. TrOCR - Microsoft's transformer-based OCR
5. Automatic Line Segmentation - Horizontal projection method
6. Web App with Tesseract.js - Production-ready for typed text

Documentation

📖 Complete Exploration Document - Detailed documentation of all approaches, technical challenges, solutions, and learnings

Key Learnings

• ✅ Successfully trained CRNN model on IAM dataset (50 epochs, 31.4MB model)
• ✅ Solved 6 technical challenges during training (CTC dimensions, Keras 3 compatibility, etc.)
• ✅ Automatic line segmentation works perfectly using horizontal projection
• ⚠️ Heavy cursive handwriting remains challenging for all tested models
• ✅ Tesseract.js (web app) works excellently for typed text and screenshots

Notebooks & Models

Training:

• notebooks/train_htr_tensorflow.ipynb - Kaggle notebook for CRNN training
• models/config.json - Model configuration and metadata

Inference:

• notebooks/htr_trocr_colab.ipynb - Google Colab notebook with TrOCR
• scripts/inference_htr.py - Local inference script for trained model

Web Demo for Typed Text:

• Live: https://abhigyan-shekhar.github.io/ocr-qa-segmentation/
• Works perfectly for screenshots, typed documents, and clear print

---

📸 Screenshots

Upload Interface

Clean, modern interface for uploading images and extracting Q&A pairs

---

Sample Input

Example riddle questions with answers - perfect for testing the system

---

Extraction Results

92% confidence extraction showing 6 Q&A pairs from 74 words

---

Q&A Pairs View

Beautifully formatted extracted questions and answers with proper separation

---

Raw Text Output

Raw extracted text showing all detected questions and answers

---

JSON Export

Structured JSON output ready for integration with other systems

---

🏗️ Architecture

┌─────────────────┐
│  Input Images   │  (Multi-page exam scans)
└────────┬────────┘
         │
         ▼
┌─────────────────────────┐
│  1. Preprocessing       │  Stitch, deskew, denoise
└────────┬────────────────┘
         │
         ▼
┌─────────────────────────┐
│  2. OCR Extraction      │  Tesseract / PaddleOCR
└────────┬────────────────┘
         │
         ▼
┌─────────────────────────┐
│  3. Feature Engineering │  Visual + text patterns
└────────┬────────────────┘
         │
         ▼
┌─────────────────────────┐
│  4. CRF Sequence Tagger │  BIO tagging: B-Q, I-Q, B-A, I-A
└────────┬────────────────┘
         │
         ▼
┌─────────────────────────┐
│  5. QA Pair Extraction  │  Group and pair Q&A
└─────────────────────────┘

---

📦 Project Structure

ocr_qa_segmentation/
├── src/                    # Core modules
│   ├── preprocessing.py    # Image preprocessing
│   ├── ocr_engine.py       # OCR wrapper (Tesseract/PaddleOCR)
│   ├── feature_extraction.py  # Feature engineering
│   ├── crf_model.py        # CRF model (sklearn-crfsuite)
│   ├── postprocessing.py   # QA pair extraction
│   └── utils.py            # Helper functions
├── scripts/                # Command-line tools
│   ├── train.py            # Train CRF model
│   ├── inference.py        # Process exam images
│   ├── annotate.py         # Create training data
│   └── quick_test.py       # System verification
├── examples/
│   └── demo.ipynb          # Jupyter notebook tutorial
├── models/                 # Saved models (.pkl files)
├── requirements.txt        # Python dependencies
└── README.md              # This file

---

🔬 How It Works

1. Why CRF (Not LLMs)?

Conditional Random Fields are perfect for this task because they:

• Model sequential dependencies between lines
• Handle noisy inputs gracefully
• Provide interpretable decisions
• Run efficiently on CPU
• Don't require massive datasets

2. Feature Engineering (No Deep Learning)

For each text line, we extract:

Feature	Purpose
indent_level	Answers often indented more than questions
vertical_gap	Large gaps indicate new questions
starts_with_q	Detects "Q1", "Question 1", etc.
fuzzy_starts_q	Handles OCR errors (Q→O, Q→0)
ends_with_punct	Questions often end with "?"
word_count	Short lines might be question numbers
prev_tag	Context from previous line

3. BIO Tagging Scheme

• B-Q: Begin Question
• I-Q: Inside Question (continuation)
• B-A: Begin Answer
• I-A: Inside Answer (continuation)
• O: Other (margins, headers)

---

🛠️ Advanced Usage

Annotate Training Data

python scripts/annotate.py \
    --image exam.jpg \
    --output data/training_data.json \
    --append

Train Custom Model

python scripts/train.py \
    --data data/training_data.json \
    --output models/custom_model.pkl \
    --val-split 0.2

Process with Visualization

python scripts/inference.py \
    --images exam1.jpg exam2.jpg \
    --model models/custom_model.pkl \
    --visualize \
    --output results.json

---

🔧 Technical Details

OCR Engine Options

• PaddleOCR (Default): Superior handwriting recognition, requires Python <3.13
• Tesseract (Fallback): Broad compatibility, works better with typed text

The system automatically uses PaddleOCR if available, otherwise falls back to Tesseract. To force Tesseract, set engine='tesseract' in OCREngine().

Performance

Metric	Value
Speed	~1 page/second (CPU)
Memory	500MB-2GB (depending on OCR engine)
Training Data	50-100 annotated pages recommended
Accuracy	~90% F1 on clean handwriting

Handling Edge Cases

✅ Multi-page splits: Stitches images before OCR
✅ Missing question numbers: Uses indentation + gaps + capitalization
✅ OCR errors: Fuzzy matching with Levenshtein distance
✅ Diagrams: Preserves bounding boxes for later extraction

---

📊 Example Output

Input: 2 exam pages with handwritten Q&A

Output (results.json):

[
  {
    "question_number": 1,
    "question": "What is the capital of France?",
    "answer": "Paris is the capital of France, located in northern France.",
    "confidence": 0.92
  },
  {
    "question_number": 2,
    "question": "Explain machine learning in your own words.",
    "answer": "Machine learning is a subset of AI that enables computers to learn from data without being explicitly programmed.",
    "confidence": 0.88
  }
]

---

🧪 Testing

Run the full test suite:

python scripts/quick_test.py

This tests:

1. CRF training on synthetic data
2. Feature extraction accuracy
3. QA pair extraction logic

---

🔧 Troubleshooting

Python 3.14+ Compatibility

If you have Python 3.14 or newer, PaddleOCR won't install. You have two options:

Option 1: Install Python 3.12 (Recommended for full handwriting support)

# macOS
brew install python@3.12
python3.12 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Option 2: Use Tesseract Only (Works but lower handwriting accuracy)

The system automatically falls back to Tesseract if PaddleOCR is unavailable. Just run:

pip install pytesseract opencv-python pillow numpy
python app.py  # Will use Tesseract automatically

Other Issues

• Import Error: Make sure virtual environment is activated: source venv/bin/activate
• Tesseract not found: Install Tesseract: brew install tesseract (macOS)
• GPU errors: PaddleOCR is configured for CPU mode, no GPU needed

---

🤝 Contributing

This is a proprietary project. See LICENSE for usage restrictions.

For collaboration inquiries, contact: abhigyan.shekhar@example.com

---

📜 License

Copyright © 2026 Abhigyan Shekhar. All Rights Reserved.

This software is proprietary. Reuse, modification, or distribution requires explicit written permission. See LICENSE for details.

---

🎓 Academic Context

This project was developed as part of an internship assignment demonstrating:

• Classical CV/ML approaches to document understanding
• Feature engineering without LLMs
• Production-ready ML system design

Key Constraint: No Large Language Models (GPT, BERT, etc.) allowed—only classical techniques.

---

🔗 Related Documents

• Technical Submission - Detailed approach and architecture
• Quick Start Guide - Fast setup instructions
• Jupyter Demo - Interactive tutorial

---

🙏 Acknowledgments

• Tesseract OCR: Open-source OCR engine
• sklearn-crfsuite: Python CRF implementation
• OpenCV: Image processing toolkit

---

Built with ❤️ using Classical Machine Learning