CaseLens-VLM

Enterprise multimodal document intelligence with VLMs, hybrid retrieval, citations, and audit controls.

CaseLens-VLM is a benchmark-oriented multimodal RAG pipeline over real scanned DocVQA pages. It uses open vision-language models to turn page images into searchable evidence, retrieves cited pages for questions, evaluates retrieval quality, and maps the same pattern to an enterprise AWS reference architecture.

The project evaluates a practical question: when documents are scanned pages, forms, tables, handwriting, and layout-heavy reports, how much does visual page understanding improve retrieval compared with metadata alone?

Why This Matters

On a 100-page / 339-question DocVQA subset, metadata-only retrieval reached 0.035 Recall@5. Qwen3-VL page evidence reached 0.658 Recall@5, and hybrid BM25 + MiniLM retrieval over the same VLM evidence reached 0.708 Recall@5.

The industry lesson is direct: for scanned enterprise documents, visual page understanding can turn otherwise weak metadata into retrievable, cited evidence. This is a document-AI analogue of evaluation-led RAG work: measure the evidence layer before optimizing generation.

Reuse Path for Builders

• Industry brief: concise VLM/document-RAG framing for AI teams.
• Real results: measured Qwen3-VL, metadata-only, and hybrid retrieval results.
• Reference architecture: publishable enterprise AWS architecture.
• Technical tradeoffs: design decisions and limitations.
• LinkedIn draft: public-facing project narrative.

No-GPU smoke check:

make portfolio-smoke

The smoke check validates the retrieval, citation, audit, and observability wiring. It does not rerun VLM inference.

Highlights

Capability	What is implemented
Real data	DocVQA scanned document pages from the UCSF Industry Documents Library
VLM evidence	Qwen3-VL page-level visual summaries
Retrieval	BM25 baseline plus MiniLM dense hybrid retrieval
Evaluation	Same 100-page / 339-question subset across all retrieval modes
Governance	Local citation audit, grounding checks, reviewer-oriented UI, AWS guardrail mapping
Infrastructure	Isambard GH200 benchmark workflow plus no-GPU portfolio demo

Model Choice

The project is standardized on Qwen3-VL-8B-Instruct as the primary open VLM. This keeps the benchmark controlled: one documented VLM evidence layer, compared against metadata-only retrieval and hybrid lexical+dense retrieval baselines.

Problem

Most enterprise RAG demos assume clean extracted text. Real document estates are messier: scanned pages, forms, charts, tables, signatures, handwriting, and visual layout cues. CaseLens-VLM shows how to convert those pages into retrievable evidence while preserving page-level provenance.

What This Does

• Runs VLM inference over document page images
• Builds page-level evidence records with provenance
• Indexes VLM evidence with lexical and hybrid retrieval
• Answers questions with cited source pages
• Evaluates retrieval with real DocVQA question-to-page labels
• Provides audit, guardrail, and AWS reference-architecture material

Dataset

The main dataset is DocVQA, a document visual question answering benchmark built from real document pages from the UCSF Industry Documents Library. Raw dataset files are not committed to this repository.

Useful references:

• DocVQA dataset: https://site.docvqa.org/datasets/docvqa
• Qwen3-VL model family: https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct

Architecture

flowchart LR
    A["DocVQA scanned pages"] --> B["Page extraction"]
    B --> C["OCR and metadata"]
    B --> D["VLM page understanding"]
    C --> E["Evidence records"]
    D --> E
    E --> F["BM25 and vector indexes"]
    F --> G["Cited question answering"]
    G --> H["Grounding audit"]
    H --> I["Retrieval evaluation"]

Loading

The publishable enterprise version is documented in docs/reference_architecture.md. Additional open-source, DialogXR-style, and AWS topology diagrams are in docs/architecture_blueprints.md.

Observability

The RAG pipeline emits Langfuse traces for retrieval and generation calls, including latency, TTFT, token usage, retrieval scores, and query/page metadata. Langfuse can run self-hosted for air-gapped deployments, with no dependency on Langfuse Cloud.

Configure tracing with:

LANGFUSE_PUBLIC_KEY=pk-lf-local
LANGFUSE_SECRET_KEY=sk-lf-local
LANGFUSE_HOST=http://localhost:3000

This mirrors enterprise observability requirements: tracing LLM calls and measuring model performance.

No-GPU Portfolio Demo

If Isambard or GPU access is unavailable, the project still has a runnable portfolio path. The benchmark results below remain the measured Qwen3-VL run; the no-GPU path is a deterministic smoke test and public demo for the retrieval, citation, audit, architecture, and observability wiring.

Run the offline smoke test:

make portfolio-smoke

This builds a tiny in-memory evidence index, runs a cited retrieval query, and verifies that Langfuse tracing stays optional when credentials are not configured. It does not run live VLM inference and does not require DocVQA images, model weights, GPU access, or Langfuse.

Run the public demo locally:

streamlit run app.py

The Streamlit demo uses bundled evidence snippets and user-provided upload notes to show the application flow without running Qwen3-VL live. A production deployment would connect the same retrieval and audit pattern to a GPU-hosted VLM, Bedrock multimodal model, or SageMaker endpoint.

Isambard Quickstart

The project assumes DocVQA has already been downloaded to:

$SCRATCH/vlm_doc_project/docvqa_hf

Prepare a 500-question sample:

cd "$SCRATCH/vlm_doc_project/caselens-vlm"
source "$HOME/miniforge3/etc/profile.d/conda.sh"
conda activate vlm-doc

pip install -r requirements.txt

python scripts/prepare_docvqa.py \
  --dataset "$SCRATCH/vlm_doc_project/docvqa_hf" \
  --out data/docvqa_sample \
  --split validation \
  --limit 500

Run the demo retrieval baseline:

python scripts/build_index.py \
  --records data/docvqa_sample/page_records.jsonl \
  --out data/docvqa_sample/index_demo.json \
  --include-gold-questions

python scripts/evaluate_retrieval.py \
  --index data/docvqa_sample/index_demo.json \
  --records data/docvqa_sample/page_records.jsonl \
  --qas data/docvqa_sample/qa_records.jsonl \
  --k 5 \
  --out data/docvqa_sample/eval_demo.json

Generate Qwen3-VL page summaries:

# Install a GPU-compatible PyTorch build first, then:
pip install -r requirements-vlm.txt

python scripts/generate_vlm_summaries.py \
  --records data/docvqa_sample/page_records.jsonl \
  --image-root data/docvqa_sample \
  --out data/docvqa_sample/vlm_qwen3_8b_100.jsonl \
  --model Qwen/Qwen3-VL-8B-Instruct \
  --max-new-tokens 180 \
  --max-pixels 501760 \
  --limit 100 \
  --resume

On shared clusters, use --local-files-only with a cached model snapshot to avoid Hugging Face rate limits.

Build a strict VLM-summary index:

python scripts/build_index.py \
  --records data/docvqa_sample/vlm_qwen3_8b_100.jsonl \
  --out data/docvqa_sample/index_qwen3_8b_100.json \
  --text-field vlm_summary

python scripts/ask.py \
  --index data/docvqa_sample/index_qwen3_8b_100.json \
  --records data/docvqa_sample/vlm_qwen3_8b_100.jsonl \
  --question "What is the PD?" \
  --k 3

To run directly from the Slurm launcher:

sbatch slurm/run_qwen3_100.sbatch

The public Streamlit demo does not require generated artifacts, but the full local artifact mode expects the Qwen3 records and index above.

Current Verified Results

The project now has real Qwen3-VL results over a DocVQA page subset. Full details are in docs/results.md.

Sample	Pages	Questions	Mode	Recall@1	Recall@5
Qwen3-VL real run	100	339	hybrid BM25 + MiniLM embeddings	not measured	0.708
Qwen3-VL real run	100	339	strict VLM-summary retrieval	0.445	0.658
Same subset	100	339	metadata-only retrieval	0.003	0.035
Same subset	100	339	demo gold-question retrieval	0.923	0.988
Smoke	29	100	demo gold-question retrieval	not measured	1.000
Main	138	500	demo gold-question retrieval	not measured	0.978

The Qwen3-VL-8B run completed in 30m42s using a capped visual token budget on an NVIDIA GH200 GPU node through the public Isambard container /lus/lfs1aip2/projects/public/u6ei/torch_cuda126.sif. The hybrid row adds a lightweight dense retrieval pass using sentence-transformers/all-MiniLM-L6-v2 over the same VLM summaries.

These are previously measured results, not regenerated by the no-GPU portfolio demo.

On Isambard, verify the PyTorch build before VLM inference:

python - << 'EOF'
import torch
print(torch.__version__)
print("cuda_available:", torch.cuda.is_available())
print("torch_cuda:", torch.version.cuda)
EOF

CLI Reference

python scripts/prepare_docvqa.py --dataset PATH --out DIR --split validation --limit 500
python scripts/generate_vlm_summaries.py --records page_records.jsonl --image-root DIR --out vlm_qwen3_8b_100.jsonl
python scripts/build_index.py --records RECORDS --out index.json --text-field metadata|vlm_summary
python scripts/ask.py --index index.json --records RECORDS --question "..." --k 5
python scripts/evaluate_retrieval.py --index index.json --records RECORDS --qas qa_records.jsonl --k 5
python scripts/audit_run.py --index index.json --records RECORDS --question "..." --answer "..." --out audit.jsonl
python scripts/portfolio_smoke.py

Streamlit Demo

The app has two modes:

• Public demo mode: works on Streamlit Community Cloud without raw DocVQA images or generated artifacts, and accepts uploaded document images as demo evidence.
• Local artifact mode: shows retrieved page images when the Isambard-generated DocVQA files are available.

Launch locally:

streamlit run app.py

The app shows verified benchmark metrics, a live retrieval demo with image upload, an enterprise architecture view, and an optional local viewer for generated records. See docs/streamlit_deploy.md for deployment notes.

For portfolio use, deploy the app on Streamlit Community Cloud from this repository with app.py as the entry point. No secrets are required for the public demo.

Repository Policy

This repo intentionally excludes:

• Raw DocVQA images and saved datasets
• Generated indexes and JSONL outputs
• Model weights and Hugging Face caches
• Slurm logs

See docs/aws_architecture.md for the AWS implementation mapping and docs/cv_project_summary.md for a concise project summary. See docs/results.md for real Qwen3-VL retrieval metrics. See docs/linkedin_post.md and docs/project_brief.md for public-facing summaries. See docs/technical_tradeoffs.md for design tradeoffs and implementation notes. See docs/model_decision.md for why the repo uses a single primary VLM. See docs/enterprise_architecture.md for the guardrails, audit, and monitoring design. See docs/reference_architecture.md for a publishable enterprise AWS reference architecture.

Limitations

• The demo baseline uses DocVQA question text and is only a pipeline sanity check.
• The strict VLM path depends on downloading and running a VLM such as Qwen3-VL-8B.
• PyTorch/CUDA installation is cluster-specific; use a build compatible with the active NVIDIA driver.
• Official DocVQA OCR transcriptions are not bundled in the Hugging Face mirror used here.