Add VLM-based Handwritten Text Recognition Pipeline (GSoC 2026 - Aniket Junghare)

[junghare-aniket]

Summary

This PR adds a VLM-centric end-to-end Handwritten Text Recognition (HTR) pipeline for early modern Spanish manuscripts, developed as part of GSoC 2026.

Unlike a traditional OCR-first approach, this pipeline places a Vision-Language Model (Qwen2.5-VL-7B-Instruct, LoRA fine-tuned) at the center of every processing stage: document analysis, literal reading, and correction, rather than using a VLM only as a late-stage corrector. A fine-tuned TrOCR model (developed during GSoC 2025) is used as an optional supplementary line-level OCR stage.

What's included

• finetune.py — LoRA fine-tuning script for Qwen2.5-VL-7B-Instruct
• inference.py — full 4-stage inference pipeline (VLM + optional TrOCR)
• evaluate.py — CER/WER evaluation script
• app/ — Gradio web app (same code deployed live on Hugging Face Spaces)
• scripts/ — SLURM job scripts for fine-tuning, inference, and evaluation
• Scoped README with setup instructions, evaluation results, and links to further resources

Results

Evaluated on the Rodrigo, Orinoco, and Tridis datasets. Full CER/WER tables and analysis are in the folder's README.

Try it live

• Live demo: https://huggingface.co/spaces/aniket-junghare/spanish-handwriting-ocr
• LoRA adapter weights: https://huggingface.co/aniket-junghare/qwen2.5-vl-spanish-ocr-lora
• Full project repo: https://github.com/junghare-aniket/vlm-handwriting-recognition-es
• Blog posts: Midterm · Final