pdf-quality-report

A small local demo for checking the quality of parsed PDF output.

It takes JSON exported by Docling and creates:

• a small traceable JSON file with text blocks, page numbers, bounding boxes, and provenance
• a deterministic Markdown quality report that highlights missing fields, broken provenance, invalid bounding boxes, and layout/noise signals
• provenance-preserving JSONL chunk records from normalized parser output

This project does not try to replace Docling. Docling does the PDF parsing. This repo checks whether the parser output is structured and traceable for human review before downstream use.

It does not claim OCR quality, RAG answer quality, table reconstruction, legal extraction, compliance readiness, production extraction accuracy, or enterprise readiness.

Technical note: the traceable output is plain JSON. It is inspired by UIF, an internal block model with text, page numbers, bounding boxes, and provenance, but no separate schema repository is needed to use this demo.

What The Report Checks

The report answers practical questions such as:

1. Does every extracted block have the basic fields a pipeline needs?
2. Can each block be traced back to a source page and bounding box?
3. Does the page contain layout noise that a human should review before using the output?

The seven current checks are:

1. Required field coverage: are core fields such as id, type, page_number, bbox, and reading_order_index present?
2. Table output structure signals: do table-labeled normalized blocks contain visible table-output structure signals?
3. Provenance completeness: can each block be traced back to Docling metadata and the source PDF?
4. BBox sanity: are bounding boxes present, finite, and ordered correctly?
5. Content vs noise ratio: how much looks like document content compared with headers, footers, images, or other secondary blocks?
6. Text usefulness: are text blocks empty, suspiciously short, or duplicated?
7. Text extraction health: is extracted text available in the normalized blocks for review, without judging whether the text is correct?

Text usefulness flags local text-quality issues such as empty body blocks, very short text, or repeated fragments. Text extraction health summarizes whether the document contains enough extracted text overall for review. The table output structure signal check flags table-labeled normalized blocks that lack obvious structured table-output signals; it does not validate table reconstruction correctness against the PDF. Structured table-output signals include normalized grid fields such as data_grid or rows, optional cells / html, or delimiter-like content.text.

Warning-only checks can move the report to REVIEW; the report CLI still exits non-zero only for hard failures.

It also lists diagnostic noise/layout signals such as table-marker artifacts, headers/footers, and ambiguous image blocks. These signals are for review only; they do not add hard failures or warnings by themselves.

Install

python -m venv .venv
. .venv/bin/activate
python -m pip install -e ".[dev]"

Docling is only needed when you want to create the parser JSON from a PDF:

python -m pip install -e ".[docling]"

Example Workflow

The example uses:

Hansen, M.; Pomp, A.; Erki, K.; Meisen, T. "Data-Driven Recognition and Extraction of PDF Document Elements." Technologies 2019, 7(3), 65. https://doi.org/10.3390/technologies7030065

Download the PDF from the official article page:

https://www.mdpi.com/2227-7080/7/3/65

Then generate Docling JSON:

docling path/to/article.pdf --to json --image-export-mode placeholder --no-ocr --output tmp/docling

Replace article.json in the next command with the JSON filename generated by Docling.

Convert selected pages to normalized JSON:

python -m pdf_quality_report.convert \
  --input tmp/docling/article.json \
  --output examples/mdpi_pdf_elements/page_006/normalized_blocks.json \
  --pages 6 \
  --source-pdf path/to/article.pdf

Create a Markdown quality report:

python -m pdf_quality_report.report \
  --input examples/mdpi_pdf_elements/page_006/normalized_blocks.json \
  --output examples/mdpi_pdf_elements/page_006/quality_report.md

Export Review Findings

Export WARN and FAIL quality-check details as a human-reviewable findings list with source context when available:

python -m pdf_quality_report.review_findings \
  --input examples/mdpi_pdf_elements/page_012/normalized_blocks.json \
  --output /tmp/page_012_review_findings.md

Use --out as a short alias for --output. See Export Review Findings for a short walkthrough.

review_findings counts WARN/FAIL detail items, or one check-level item when a warning or failure only has metric-style details. It is not the number of WARN/FAIL checks. The command exits with 0 when the Markdown file is written successfully, even if review findings exist; input or output errors return 1.

If the package entry point is installed:

pdf-quality-review-findings \
  --input examples/mdpi_pdf_elements/page_012/normalized_blocks.json \
  --output /tmp/page_012_review_findings.md

Export Review Bundle

Export a static folder with the quality report, review findings, clean Markdown, chunk records, and chunk review Markdown:

python -m pdf_quality_report.review_bundle \
  --input examples/mdpi_pdf_elements/page_012/normalized_blocks.json \
  --output-dir /tmp/page_012_pqr_review_bundle

The command exits with 0 when all bundle files are written, even if the report decision is REVIEW or BLOCK. Input or output errors return 1. See Export Review Bundle for a short walkthrough.

Use --out-dir as a short alias for --output-dir.

If the package entry point is installed:

pdf-quality-review-bundle \
  --input examples/mdpi_pdf_elements/page_012/normalized_blocks.json \
  --output-dir /tmp/page_012_pqr_review_bundle

Export Clean Markdown

Export normalized blocks to clean Markdown with source reference comments for downstream review or retrieval-oriented preparation:

python -m pdf_quality_report.to_markdown \
  --input examples/mdpi_pdf_elements/page_012/normalized_blocks.json \
  --output /tmp/page_012.md

Use --out as a short alias for --output. See Export Clean Markdown for a short walkthrough.

If the package entry point is installed:

pdf-quality-markdown \
  --input examples/mdpi_pdf_elements/page_012/normalized_blocks.json \
  --output /tmp/page_012.md

Export Chunk Records

Export normalized blocks to JSONL chunk records with block IDs, page numbers, bounding-box references, heading context, source metadata, and citation-style display strings preserved for downstream review or lightweight retrieval-oriented experiments:

python -m pdf_quality_report.chunk \
  --input examples/mdpi_pdf_elements/page_006/normalized_blocks.json \
  --output /tmp/page_006_chunks.jsonl

Use --out as a short alias for --output. See Export Chunk Records for a short walkthrough.

If the package entry point is installed:

pdf-quality-chunk \
  --input examples/mdpi_pdf_elements/page_006/normalized_blocks.json \
  --output /tmp/page_006_chunks.jsonl

Export Chunk Review Markdown

Build and render chunk records from normalized blocks as human-reviewable Markdown with source references and heading context:

python -m pdf_quality_report.chunk_review \
  --input examples/mdpi_pdf_elements/page_006/normalized_blocks.json \
  --output /tmp/page_006_chunk_review.md

Use --out as a short alias for --output. See Export Chunk Review Markdown for a short walkthrough.

If the package entry point is installed:

pdf-quality-chunk-review \
  --input examples/mdpi_pdf_elements/page_006/normalized_blocks.json \
  --output /tmp/page_006_chunk_review.md

Included Examples

This repository includes derived example outputs for page 6 and page 12 of the sample article:

• examples/mdpi_pdf_elements/page_006/normalized_blocks.json
• examples/mdpi_pdf_elements/page_006/quality_report.md
• examples/mdpi_pdf_elements/page_012/normalized_blocks.json
• examples/mdpi_pdf_elements/page_012/quality_report.md

The source PDF itself is not committed. See examples/mdpi_pdf_elements/SOURCE.md for attribution and license details.

The tests use small synthetic Docling-style JSON fixtures, not the committed MDPI example files.

Canonical visuals

Documentation assets under docs/ explain the pipeline and a sample report layout. They do not change runtime behavior or widen claims beyond this README (see Publication policy).

• docs/pqr_pipeline.svg — flow from PDF input through Docling to normalized blocks, six checks, GO/REVIEW/BLOCK, and exports
• docs/pqr_before_after.svg — illustrative raw parser JSON versus structured quality-report view
• docs/sample_report.html — open locally in a browser for a formatted sample (page 6 example)

Portfolio layer (same facts, presentation framing for proposals and embeds): docs/portfolio/README.md.

Example Report Output Excerpt

The simplified excerpt below comes from page 12 of the included MDPI research-paper example. A block is one extracted page element, such as text, a header, an image, or a caption. Later check sections and block-level warning details are not shown.

## Summary
- total_blocks: 82
- hard_failures: 0
- warnings: 2
- decision: REVIEW

## Interpretation
Why REVIEW? The hard structure checks passed, but warning checks need review in this report: Content vs Noise Ratio and Text Usefulness. Warnings are decision-level findings. Noise / Layout Signals provide supporting evidence and are counted separately.

- Content vs Noise Ratio is WARN: The report treats 79 blocks as possible main document text. The report treats 3 blocks as possible layout or noise elements.
- 2 blocks are typed as headers: p12-texts-555 and p12-texts-556. 1 block is typed as image: p12-pictures-14.
- Text Usefulness is WARN: This report found 22 very short extracted text values and 6 repeated-text groups.
- These findings come from extracted text blocks. Quoted values are the actual extracted text values, and block IDs show where they came from in the normalized output.
- In this report, very short means normalized text with 3 characters or fewer.
- Very short numeric fragments such as '111', '0.5', and '225' are common in chart ticks, figure labels, or repeated headers, footers, or page labels.

## Noise / Layout Signals
These signals identify page-layout elements that may need review before reusing the extracted text. They are supporting evidence and are counted separately from warning checks.

- table_marker_artifacts: 0
- running_furniture_blocks: 2
- visual_anchor_blocks: 1
- ambiguous_image_blocks: 1

REVIEW means the hard structure checks passed, but warning details still need human review before Markdown export, retrieval-oriented preparation, or manual extraction.

License

Code in this repository is MIT licensed.

The sample article is third-party content distributed under CC BY 4.0 by MDPI. Derived example artifacts are provided for demonstration with attribution; they are not endorsed by MDPI or the article authors.