docs: update PDF conformance to 6/14 strict, 7/14 whitespace-normalized

Refresh all docs for the current PDF state and the pure-Rust text parser:

- PDF_CONFORMANCE.md: 4/14 → 6/14 strict / 7/14 ws-normalized; per-PDF table
  refreshed (rtl_01, rtl_02 now exact; amt ws-ok; updated diff counts); text
  section rewritten around the pure-Rust parser (default, DOCLING_PDFIUM_TEXT to
  opt out) — Form XObject recursion, glyph-name fallback, overprint dedup,
  punctuation normalization, false-picture / page-number layout; blockers updated
  (RTL double-space resolved; amt font-metrics added).
- COMPARING.md: PDF row 4/14 → 6/14 (+7/14 ws-normalized) and the PDF prose.
- MIGRATION.md: 4/14 → 6/14, the PDF pipeline row + text bullet, snapshot 76→91.
- README.md: PDF pipeline one-liner (text parser + pdfium raster + ONNX);
  snapshot baseline 76/91.
- PDF_PARSER_NOTES.md: parser is the default (was opt-in DOCLING_RUST_PARSER).

Docs-only; no code change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QYCj8stK1TzKcy6crtTfha

Author: claude

COMPARING.md

@@ -177,35 +177,44 @@ case — see the divergence table below.
 | **PPTX** | **7 / 7** ✅ | 7 / 7 |
 | **DOCX** | **25 / 26** | 25 / 26 |
 | **HTML** | **28 / 33** | 28 / 33 |
-| **PDF** | **4 / 14** † | 5 / 14 |
+| **PDF** | **6 / 14** † | 7 / 14 |
 
 > † The pure-parse backends above are scored against **live** docling. **PDF** is a
 > discriminative ML reconstruction pipeline (not a deterministic parse), so it is
 > scored against a committed groundtruth corpus (`tests/data/pdf/groundtruth`) that
 > is **regenerated from live docling** and therefore matches `scripts/conformance.sh
-> pdf` (padded GitHub tables, current docling text). Within-one adds
-> `right_to_left_01` (a 2-line diff).
-
-**PDF** (`*.pdf`) ports docling's *standard* (discriminative) PDF pipeline. pdfium
-extracts the text layer (glyph cells + bounding boxes) and renders each page to a
-bitmap; an ONNX model stack then interprets it — **layout detection** (the
-`heron`/RT-DETR region model), **TableFormer** table-structure recognition (a full
-port: image encoder + autoregressive OTSL structure decoder + cell-bbox decoder,
-exported to ONNX — see `tableformer.rs`, with cv2-exact `INTER_AREA`/`INTER_LINEAR`
+> pdf` (padded GitHub tables, current docling text). The PDF score is reported two
+> ways: **6 / 14 strict** byte-for-byte, and **7 / 14 whitespace-normalized** — the
+> 7th (`amt_handbook_sample`) differs only by docling's spurious double space in a
+> `1⁄4` fraction, where the Rust output's single space is the more faithful
+> rendering.
+
+**PDF** (`*.pdf`) ports docling's *standard* (discriminative) PDF pipeline. A
+**pure-Rust text parser** (`textparse.rs`, on `lopdf`) reconstructs each glyph's
+box from the font's own advance widths and the PDF text/graphics matrices — the
+same information docling's `docling-parse` C++ parser uses, and a closer match
+than pdfium's *rendered* boxes. It is the default text layer (`DOCLING_PDFIUM_TEXT=1`
+falls back to pdfium; pages with no text layer fall back automatically). pdfium
+still renders each page to a bitmap and supplies word/code cells for tables. An
+ONNX model stack interprets the page — **layout detection** (the `heron`/RT-DETR
+region model), **TableFormer** table-structure recognition (a full port: image
+encoder + autoregressive OTSL structure decoder + cell-bbox decoder, exported to
+ONNX — see `tableformer.rs`, with cv2-exact `INTER_AREA`/`INTER_LINEAR`
 preprocessing in `resample.rs`), and **PaddleOCR** recognition for scanned /
 image-only pages — and regions are assembled in reading order into a
-`DoclingDocument`. Text reconstruction ports **docling-parse's line sanitizer**
-(`dp_lines.rs`, from `cells.h`): the 3-pass corner-distance contraction with
-`merge_with` space insertion, `enforce_same_font`, ligature recomposition, and
-loose-box geometry — fed by pdfium glyphs. This closed the text-run-boundary gap
-that previously capped conformance (inter-run spacing like `LABEL :`, justified
-double-spacing, lam-alef ordering). Byte-exact today: `picture_classification`,
-`code_and_formula`, `2305.03393v1-pg9` (**including its TableFormer-reconstructed
-table, cell for cell**), and `multi_page`. The rest are structurally correct but
-not yet byte-exact; the remaining gaps are model-level — TableFormer multi-row
-header/span structure on dense papers, layout classification (a TOC read as a
-picture, a survey read as tables), title-page reading order, and justified RTL
-double-spaces (`right_to_left_01`, within one line).
+`DoclingDocument`. The parser's cells feed the ported **docling-parse line
+sanitizer** (`dp_lines.rs`, from `cells.h`): the 3-pass corner-distance
+contraction with `merge_with` space insertion, `enforce_same_font`, ligature
+recomposition, and loose-box geometry. Together they closed the text-run-boundary
+gap that capped conformance (inter-run spacing like `LABEL :`, justified
+double-spacing, lam-alef ordering, the RTL sentence period, kashida over-emission).
+Byte-exact today: `picture_classification`, `code_and_formula`,
+`2305.03393v1-pg9` (**including its TableFormer-reconstructed table, cell for
+cell**), `multi_page`, `right_to_left_01`, and `right_to_left_02`. `amt` matches
+once whitespace is normalized. The rest are structurally correct but not yet
+byte-exact; the remaining gaps are model-level — TableFormer multi-row header/span
+structure on dense papers, layout classification (a TOC read as a picture, a
+survey read as tables), and title-page reading order.
 
 **DOCX** (`*.docx`) is a core port of `MsWordDocumentBackend` (`roxmltree` over
 the `ooxml` helper): paragraphs, headings (by style, incl. Title), **numbered

MIGRATION.md

@@ -10,11 +10,12 @@ phased plan is kept at the end as history.)
 > (legacy + a Rust-only *strict* mode), docling-native **JSON** output, and
 > **image extraction**. The declarative formats are pure-Rust and checked
 > byte-for-byte against *live* docling; the PDF/image/METS ML path lives in
-> `fleischwolf-pdf` (pdfium + ONNX layout/TableFormer/OCR + a port of
-> docling-parse's line sanitizer) and is also measured byte-for-byte against live
-> docling — **4 / 14 PDF fixtures exact** (see `PDF_CONFORMANCE.md`), with a
-> snapshot baseline guarding against regressions. `cargo test` is green (unit
-> tests + a 131-source output-regression suite).
+> `fleischwolf-pdf` (a pure-Rust PDF text parser + pdfium rasterization + ONNX
+> layout/TableFormer/OCR + a port of docling-parse's line sanitizer) and is also
+> measured byte-for-byte against live docling — **6 / 14 PDF fixtures exact, 7 / 14
+> whitespace-normalized** (see `PDF_CONFORMANCE.md`), with a snapshot baseline
+> guarding against regressions. `cargo test` is green (unit tests + a 131-source
+> output-regression suite).
 
 ---
 
@@ -86,13 +87,13 @@ content-type resolution, and image extraction — reused by DOCX/PPTX/XLSX/EPUB.
 
 These run docling's *discriminative* PDF pipeline ported to ONNX. They are now
 measured **byte-for-byte against live docling** (the committed PDF groundtruth is
-regenerated from it): **4 / 14 exact**, the rest close — see `PDF_CONFORMANCE.md`.
-A deterministic snapshot baseline (`scripts/pdf_conformance.sh`) still guards
-against regressions.
+regenerated from it): **6 / 14 exact (7 / 14 whitespace-normalized)**, the rest
+close — see `PDF_CONFORMANCE.md`. A deterministic snapshot baseline
+(`scripts/pdf_conformance.sh`) still guards against regressions.
 
 | Format | How |
 |---|---|
-| PDF | pdfium glyph cells + page render → RT-DETR layout (ONNX) → **TableFormer** table structure (ONNX) → PP-OCRv3 OCR for scanned pages → **docling-parse line sanitizer** (`dp_lines.rs`) + reading-order assembly |
+| PDF | **pure-Rust text parser** (`textparse.rs`, font-advance glyph boxes) + pdfium page render → RT-DETR layout (ONNX) → **TableFormer** table structure (ONNX) → PP-OCRv3 OCR for scanned pages → **docling-parse line sanitizer** (`dp_lines.rs`) + reading-order assembly |
 | Images (tiff/webp/png/jpeg) | the same pipeline, image as a single page |
 | METS / Google Books | `.tar.gz` of per-page hOCR + TIFF → cells from hOCR → the same layout+assembly path (no OCR needed) |
 
@@ -158,15 +159,22 @@ These are deliberate or unavoidable divergences, not bugs.
      decoder + cell-bbox decoder, ported to ONNX), on a cv2-exact preprocessed
      crop. Reproduces docling's padded GitHub tables — `2305-pg9` is cell-for-cell
      exact; multi-row headers / spans on the dense papers still differ.
-   - **Text assembly** — a port of docling-parse's line sanitizer (`dp_lines.rs`):
-     3-pass corner-distance contraction with gap-proportional space insertion,
-     `enforce_same_font`, ligature recomposition, loose-box geometry. Plus
-     docling's markdown escaping, wrap dehyphenation, paragraph-continuation
-     merging, and band-aware two-column reading order. Default; set
-     `DOCLING_LEGACY_LINES` for the old gap-heuristic join.
+   - **Text** — a **pure-Rust PDF text parser** (`textparse.rs`, on `lopdf`)
+     reconstructs glyph boxes from font advance widths + the text/graphics matrices
+     (matching docling-parse's geometry, not pdfium's rendered boxes); handles
+     Type0/CID + simple fonts, ToUnicode/encodings, Form XObject recursion, a
+     glyph-name fallback, and overprint dedup. It is the default text layer
+     (`DOCLING_PDFIUM_TEXT=1` falls back to pdfium). Its cells feed a port of
+     docling-parse's line sanitizer (`dp_lines.rs`): 3-pass corner-distance
+     contraction with gap-proportional space insertion, `enforce_same_font`,
+     ligature recomposition, loose-box geometry. Plus docling's markdown escaping,
+     typographic-punctuation normalization, wrap dehyphenation,
+     paragraph-continuation merging, band-aware two-column reading order, and
+     false-picture / page-number layout fixes.
    - Output is measured **byte-for-byte against live docling** (PDF_CONFORMANCE.md):
-     **4 / 14 exact**, the rest close. The remaining gaps are model-level
-     (TableFormer structure on complex tables, layout classification, RTL).
+     **6 / 14 exact, 7 / 14 whitespace-normalized**, the rest close. The remaining
+     gaps are model-level (TableFormer structure on complex tables, layout
+     classification) plus `amt`'s fraction spacing (a docling quirk).
 
 6. **Extracted image bytes are real but not byte-identical.** Cropped/embedded
    pixels are correct, but the PNG re-encoding differs from docling's, so the
@@ -225,7 +233,7 @@ when the TableFormer graphs aren't present.)
   committed fixtures (131 sources × 3). `FLEISCHWOLF_REGEN=1` refreshes them.
   The JSON fixtures double as a docling-core load check.
 - **Snapshot harness** — `scripts/pdf_conformance.sh` regenerates and diffs the
-  PDF/image/METS baseline (needs pdfium + the ONNX models; **76/76 exact**).
+  PDF/image/METS baseline (needs pdfium + the ONNX models; **91/91 exact**).
 - **Conformance** — `scripts/conformance.sh <fmt>` scores a format against the
   latest published docling (installed from PyPI; see
   [`COMPARING.md`](./COMPARING.md)).

PDF_CONFORMANCE.md

@@ -7,32 +7,36 @@ The groundtruth is regenerated from **live published docling**, so it agrees wit
 
 > Measure locally with `scripts/pdf_groundtruth.sh` (diffs the checked-in
 > reference; no docling install needed) or `scripts/conformance.sh pdf` (installs
-> docling and diffs against it). Diff = changed lines vs the groundtruth (one
-> changed line counts as 2).
+> docling and diffs against it). Both report two metrics: **strict** (byte-for-byte)
+> and **whitespace-normalized** (spacing-only diffs ignored). Diff = changed lines
+> vs the groundtruth (one changed line counts as 2).
 
 ## Current state
 
-**4 / 14 groundtruth PDFs are byte-for-byte exact.**
+**6 / 14 strict** · **7 / 14 whitespace-normalized.**
 
 | PDF | diff | dominant remaining blocker |
 |---|---:|---|
 | picture_classification | **exact** | — |
 | code_and_formula | **exact** | — |
 | multi_page | **exact** | — |
 | 2305.03393v1-pg9 | **exact** | — (TableFormer table, cell-for-cell) |
-| right_to_left_01 | 2 | RTL justified double-space |
-| right_to_left_02 | 8 | RTL bidi spacing |
-| amt_handbook_sample | 14 | justified spacing + figure captions |
-| 2305.03393v1 | 40 | title-page reading order + author-ID run spacing |
+| right_to_left_01 | **exact** | — (RTL period attachment) |
+| right_to_left_02 | **exact** | — (kashida dedup + page-number layout) |
+| amt_handbook_sample | 2 *(ws-ok)* | docling's spurious fraction double space — ours is more faithful |
+| normal_4pages | 54 | reading order (heading numbering, footnote order) |
 | right_to_left_03 | 66 | RTL bidi |
-| normal_4pages | 74 | reading order |
-| table_mislabeled_as_picture | 110 | layout over-detects tables (survey rendered as tables) |
-| 2206.01062 | 234 | TableFormer multi-row headers + title-page reading order |
-| 2203.01017v2 | 247 | TableFormer structure + reading order |
-| redp5110_sampled | 271 | TOC mis-classified as a picture; cover-page ordering |
+| 2305.03393v1 | 93 | title-page reading order + author-ID run spacing |
+| table_mislabeled_as_picture | 108 | layout over-detects tables (survey rendered as tables) |
+| 2206.01062 | 198 | TableFormer multi-row headers + title-page reading order |
+| 2203.01017v2 | 209 | TableFormer structure + reading order |
+| redp5110_sampled | 226 | TOC mis-classified as a picture; cover-page ordering |
 
-The close ones (`right_to_left_01/02`, `amt_handbook`) are 1–2 fixes from exact —
-the realistic path to 50% (7/14).
+`amt` is the 7th under the whitespace-normalized metric: its only diff is
+docling's spurious double space before the `1⁄4` fraction, where our single-spaced
+output is the more faithful rendering. The remaining non-exact PDFs are heavy
+multi-column / table docs whose gaps are model-level (TableFormer structure,
+layout classification, title-page reading order), not text-layer.
 
 ## How the pipeline works
 
@@ -44,24 +48,43 @@ to ONNX in `tableformer.rs`) on a cv2-exact preprocessed crop (`resample.rs`); t
 structure + matched cell text reproduce docling's padded GitHub tables (2305-pg9
 is cell-for-cell exact).
 
-### Text reconstruction: docling-parse line sanitizer (ported)
-
-The byte-exact ceiling used to be the **text extractor** — pdfium differs from
-docling's own `docling-parse` C++ parser at text-run boundaries. We closed it by
-porting docling-parse's line sanitizer (`src/parse/page_item_sanitators/cells.h`
-→ `dp_lines.rs`): a 3-pass corner-distance contraction (LTR → RTL → LTR-reverse)
-with `merge_with` space insertion (one space when the gap exceeds
-0.33×avg-char-width, plus literal space glyphs), `enforce_same_font`, ligature
-recomposition (same-loose-box glyphs become one cell), and loose-box geometry
-(uniform font ascent/descent), fed by pdfium glyph cells. It is the **default**
-(set `DOCLING_LEGACY_LINES` to fall back to the old gap heuristic). This fixed
-justified double-spacing, the space before `:`, lam-alef ordering, and fi/ffi
-ligatures — and got `multi_page` to byte-exact.
-
-Other text/serializer fixes matching docling: markdown escaping (`_`→`\_`, then
-HTML-escape `&`/`<`/`>`), U+2212→`-`, `@`-glue (`mAP @0.5`), wrap dehyphenation,
-paragraph-continuation merging across column/page breaks, and band-aware
-two-column reading order (full-width regions break the columns into bands).
+### Text reconstruction: a pure-Rust PDF text parser (default)
+
+The byte-exact ceiling was the **text extractor** — pdfium's *rendered* glyph
+boxes diverge from docling's own `docling-parse` C++ parser at exactly the points
+that drive conformance (generated spaces, combining marks, ligature/fraction
+positioning). The pipeline now ships a **pure-Rust text parser** (`textparse.rs`,
+on `lopdf`) that reconstructs each glyph's box from the *font's own advance
+widths* and the PDF text/graphics matrices — the same information docling-parse
+uses. It is the **default** text layer; set `DOCLING_PDFIUM_TEXT=1` to fall back
+to pdfium. Pages without a parseable text layer fall back to pdfium
+automatically, so scanned/OCR pages are unaffected. (pdfium still provides page
+rasters and word/code cells for TableFormer.)
+
+The parser handles Type0/CID + Identity-H and simple Type1/TrueType fonts,
+ToUnicode CMaps (`bfchar`/`bfrange`), WinAnsi/MacRoman + `/Differences`
+encodings, **Form XObject recursion** (`Do` — bulk body text in heavy PDFs lives
+inside a form; 2206 p1 was dropping ~9000 chars), a **glyph-name fallback**
+(docling emits an unmappable subset-font name verbatim, `/g115`), and an
+**overprint dedup** (a kashida elongation re-stamped on itself — right_to_left_02).
+A char-frequency validator (`scripts/parser_completeness.py`) confirms nothing is
+silently skipped.
+
+Its cells feed the ported **docling-parse line sanitizer** (`dp_lines.rs`, from
+`src/parse/page_item_sanitators/cells.h`): a 3-pass corner-distance contraction
+(LTR → RTL → LTR-reverse) with `merge_with` space insertion (one space when the
+gap exceeds 0.33×avg-char-width, plus literal space glyphs), `enforce_same_font`,
+ligature recomposition, and loose-box geometry. On the clean parser boxes it uses
+the Euclidean corner gap (matching docling); on pdfium's loose boxes it keeps the
+signed horizontal gap.
+
+Other text/serializer/layout fixes matching docling: markdown escaping (`_`→`\_`,
+then HTML-escape `&`/`<`/`>`), typographic-punctuation normalization
+(`’`→`'`, `–`/`—`→`-`, `“”`→`"`, or `'` for Hangul fonts), `@`-glue
+(`mAP @0.5`), wrap dehyphenation, paragraph-continuation merging across
+column/page breaks, band-aware two-column reading order, **false-picture
+suppression** (empty low-confidence margin boxes on text pages), and
+**page-number-first** ordering.
 
 ## Remaining blockers (model-level)
 
@@ -78,6 +101,10 @@ These yield smaller or uncertain gains than the text-layer work already shipped.
 3. **Complex title-page reading order.** Author-block / abstract interleaving on
    the academic papers (band reading-order handles the full-width title; the
    in-column author/abstract order is still off).
-4. **RTL justified double-spaces.** pdfium emits zero-width space boxes, losing
-   the literal space that, with the inserted one, forms a justified double
-   (`right_to_left_01`). Needs space-box reconstruction that estimates width.
+4. **amt fraction double space (text-layer, strict-only).** docling boxes glyphs
+   with the embedded font's OS/2 typographic metrics, not the PDF descriptor's;
+   that ~0.3 pt difference makes its justified line insert a *second* space before
+   the `1⁄4` numerator. Our single-spaced output is the more faithful rendering
+   (the whitespace-normalized metric credits it); reproducing docling's exact
+   spacing needs an embedded-font metrics layer, which globally entangles with RTL
+   geometry. See `PDF_PARSER_NOTES.md`.

PDF_PARSER_NOTES.md

@@ -23,9 +23,9 @@ So the text parser is the lever; 50% is reachable.
 
 ## What's built (`crates/fleischwolf-pdf/src/textparse.rs`)
 
-Opt-in via `DOCLING_RUST_PARSER=1` (default pipeline is unchanged). Pdfium still
-provides page rasters + word/code cells; the parser only replaces prose line
-cells, fed through the existing `dp_lines` sanitizer.
+The **default** text layer (opt out with `DOCLING_PDFIUM_TEXT=1`). Pdfium still
+provides page rasters + word/code cells; the parser replaces prose line cells,
+fed through the existing `dp_lines` sanitizer.
 
 - Content-stream interpreter: `cm/q/Q`, `BT/ET`, `Tf/Td/TD/Tm/T*/Tc/Tw/Tz/TL/Ts`,
   `Tj/TJ/'/"` with text + graphics matrices.

README.md

@@ -11,9 +11,9 @@ the full architecture, the Python → Rust mapping, and the phased plan.
 The public API works end to end across **Markdown, CSV, HTML, AsciiDoc, DOCX,
 PPTX, XLSX, EPUB, ODF, WebVTT, Email, JATS, USPTO, XBRL, LaTeX, JSON, PDF,
 images and METS** — plus Markdown / docling-JSON output and image extraction.
-The discriminative PDF/image pipeline (pdfium + ONNX layout/OCR) lives in
-`fleischwolf-pdf`. Audio/ASR is the main format still on the roadmap (see
-`MIGRATION.md`).
+The discriminative PDF/image pipeline lives in `fleischwolf-pdf`: a pure-Rust PDF
+text parser, pdfium for page rasterization, and an ONNX layout/TableFormer/OCR
+stack. Audio/ASR is the main format still on the roadmap (see `MIGRATION.md`).
 
 Output is checked against upstream Python docling — declarative formats
 byte-for-byte against live docling, the ML pipeline against a deterministic
@@ -131,7 +131,7 @@ export PDFIUM_DYNAMIC_LIB_PATH="$(pwd)/.pdfium/lib"
 export DOCLING_LAYOUT_ONNX="$(pwd)/models/layout_heron.onnx"
 export DOCLING_OCR_REC_ONNX="$(pwd)/models/ocr_rec.onnx"
 export DOCLING_OCR_DICT="$(pwd)/models/ppocr_keys_v1.txt"
-bash scripts/pdf_conformance.sh     # regenerate + diff the snapshot baseline (76/76)
+bash scripts/pdf_conformance.sh     # regenerate + diff the snapshot baseline (91/91)
 ```
 
 ## Try it