PDF pipeline: docling-parse line sanitizer + TableFormer + conformance push (4/14 exact)

COMPARING.md

@@ -176,7 +176,36 @@ case — see the divergence table below.
 | **XLSX** | **9 / 9** ✅ | 9 / 9 |
 | **PPTX** | **7 / 7** ✅ | 7 / 7 |
 | **DOCX** | **25 / 26** | 25 / 26 |
-| **HTML** | **28 / 32** | 29 / 32 |
+| **HTML** | **28 / 33** | 28 / 33 |
+| **PDF** | **4 / 14** † | 5 / 14 |
+
+> † The pure-parse backends above are scored against **live** docling. **PDF** is a
+> discriminative ML reconstruction pipeline (not a deterministic parse), so it is
+> scored against a committed groundtruth corpus (`tests/data/pdf/groundtruth`) that
+> is **regenerated from live docling** and therefore matches `scripts/conformance.sh
+> pdf` (padded GitHub tables, current docling text). Within-one adds
+> `right_to_left_01` (a 2-line diff).
+
+**PDF** (`*.pdf`) ports docling's *standard* (discriminative) PDF pipeline. pdfium
+extracts the text layer (glyph cells + bounding boxes) and renders each page to a
+bitmap; an ONNX model stack then interprets it — **layout detection** (the
+`heron`/RT-DETR region model), **TableFormer** table-structure recognition (a full
+port: image encoder + autoregressive OTSL structure decoder + cell-bbox decoder,
+exported to ONNX — see `tableformer.rs`, with cv2-exact `INTER_AREA`/`INTER_LINEAR`
+preprocessing in `resample.rs`), and **PaddleOCR** recognition for scanned /
+image-only pages — and regions are assembled in reading order into a
+`DoclingDocument`. Text reconstruction ports **docling-parse's line sanitizer**
+(`dp_lines.rs`, from `cells.h`): the 3-pass corner-distance contraction with
+`merge_with` space insertion, `enforce_same_font`, ligature recomposition, and
+loose-box geometry — fed by pdfium glyphs. This closed the text-run-boundary gap
+that previously capped conformance (inter-run spacing like `LABEL :`, justified
+double-spacing, lam-alef ordering). Byte-exact today: `picture_classification`,
+`code_and_formula`, `2305.03393v1-pg9` (**including its TableFormer-reconstructed
+table, cell for cell**), and `multi_page`. The rest are structurally correct but
+not yet byte-exact; the remaining gaps are model-level — TableFormer multi-row
+header/span structure on dense papers, layout classification (a TOC read as a
+picture, a survey read as tables), title-page reading order, and justified RTL
+double-spaces (`right_to_left_01`, within one line).
 
 **DOCX** (`*.docx`) is a core port of `MsWordDocumentBackend` (`roxmltree` over
 the `ooxml` helper): paragraphs, headings (by style, incl. Title), **numbered
@@ -218,9 +247,10 @@ the drawing parts via the shared `ooxml` zip/rels helper). The one miss
 all-empty table.
 
 CSV, Markdown, AsciiDoc, and the DeepSeek-OCR Markdown variant are fully
-one-to-one. HTML's 4 remaining misses all require docling-internal subsystems
-that depend on **headless-browser rendering** or are otherwise impractical to
-port — see below. 28/32 is effectively the ceiling for a pure-parse port.
+one-to-one. HTML's 5 remaining misses are a tail of docling-internal behaviours —
+some requiring **headless-browser rendering**, others (a large Wikipedia page,
+key-value form extraction) needing substantial structural work — see below. 28/33
+is roughly the ceiling for a pure-parse port.
 
 **AsciiDoc** (`*.asciidoc`/`*.adoc`) ports docling's line-oriented
 `AsciiDocBackend`: titles/sections, nested bullet/numbered lists (all rendered

MIGRATION.md

@@ -10,8 +10,11 @@ phased plan is kept at the end as history.)
 > (legacy + a Rust-only *strict* mode), docling-native **JSON** output, and
 > **image extraction**. The declarative formats are pure-Rust and checked
 > byte-for-byte against *live* docling; the PDF/image/METS ML path lives in
-> `fleischwolf-pdf` and is checked against a deterministic snapshot baseline.
-> `cargo test` is green (unit tests + a 131-source output-regression suite).
+> `fleischwolf-pdf` (pdfium + ONNX layout/TableFormer/OCR + a port of
+> docling-parse's line sanitizer) and is also measured byte-for-byte against live
+> docling — **4 / 14 PDF fixtures exact** (see `PDF_CONFORMANCE.md`), with a
+> snapshot baseline guarding against regressions. `cargo test` is green (unit
+> tests + a 131-source output-regression suite).
 
 ---
 
@@ -79,15 +82,17 @@ PyPI; run via `scripts/conformance.sh <fmt>`), not the committed groundtruth
 Shared OOXML infrastructure (`ooxml.rs`): a `zip` reader, `.rels` parsing, part
 content-type resolution, and image extraction — reused by DOCX/PPTX/XLSX/EPUB.
 
-### ML formats — `fleischwolf-pdf`, snapshot baseline
+### ML formats — `fleischwolf-pdf`
 
-These run docling's *discriminative* PDF pipeline ported to ONNX. Output is **not
-byte-for-byte** with docling (different OCR/table models — §4); it is pinned by a
-deterministic snapshot (`scripts/pdf_conformance.sh`, **76/76 exact**).
+These run docling's *discriminative* PDF pipeline ported to ONNX. They are now
+measured **byte-for-byte against live docling** (the committed PDF groundtruth is
+regenerated from it): **4 / 14 exact**, the rest close — see `PDF_CONFORMANCE.md`.
+A deterministic snapshot baseline (`scripts/pdf_conformance.sh`) still guards
+against regressions.
 
 | Format | How |
 |---|---|
-| PDF | pdfium text cells + page render → RT-DETR layout (ONNX) → PP-OCRv3 OCR for scanned pages → geometric table reconstruction → reading-order assembly |
+| PDF | pdfium glyph cells + page render → RT-DETR layout (ONNX) → **TableFormer** table structure (ONNX) → PP-OCRv3 OCR for scanned pages → **docling-parse line sanitizer** (`dp_lines.rs`) + reading-order assembly |
 | Images (tiff/webp/png/jpeg) | the same pipeline, image as a single page |
 | METS / Google Books | `.tar.gz` of per-page hOCR + TIFF → cells from hOCR → the same layout+assembly path (no OCR needed) |
 
@@ -135,22 +140,33 @@ These are deliberate or unavoidable divergences, not bugs.
    entity re-escaping); `strict` produces cleaner Markdown. docling has no such
    switch. All conformance numbers are measured in **legacy** mode.
 
-4. **Tables match *current* docling, not the committed fixtures.** docling-core's
-   Markdown table serializer emits padded GitHub tables today; the repo's
-   committed groundtruth `.md` corpus predates that and uses a minimal `| - |`
-   format. `fleischwolf` matches the **current/live** output — so table-bearing
-   formats look correct against live docling and "wrong" against the stale `.md`.
+4. **Tables use docling-core's padded GitHub format.** All backends emit the
+   width-padded `tabulate(tablefmt="github")` tables that current published
+   docling produces (columns padded to header-width+2 or the widest data cell,
+   numeric columns right-aligned). The PDF groundtruth was regenerated from live
+   docling to match. (An earlier compact `| - |` variant — to match a stale
+   committed corpus — was reverted; the `compact_tables` option still exists but
+   no backend sets it.)
 
-5. **The PDF pipeline is discriminative and partial.** Ported from docling's
-   standard pipeline, with substitutions:
+5. **The PDF pipeline is discriminative and byte-measured.** Ported from
+   docling's standard pipeline:
    - **Layout** — RT-DETR (`docling-layout-heron`) exported to ONNX, run via
      `ort`. Same model family as docling.
    - **OCR** — PP-OCRv3 recognition (RapidOCR) via ONNX, *not* docling's default
      EasyOCR; different recognizer → different scanned text.
-   - **Tables** — *geometric* grid reconstruction (cluster cells into rows/cols),
-     **not TableFormer** (docling's autoregressive table-structure model). Table
-     structure is approximate; complex spans are not recovered.
-   - Output is therefore a **snapshot baseline**, never byte-for-byte with docling.
+   - **Tables** — **TableFormer** (image encoder + autoregressive OTSL structure
+     decoder + cell-bbox decoder, ported to ONNX), on a cv2-exact preprocessed
+     crop. Reproduces docling's padded GitHub tables — `2305-pg9` is cell-for-cell
+     exact; multi-row headers / spans on the dense papers still differ.
+   - **Text assembly** — a port of docling-parse's line sanitizer (`dp_lines.rs`):
+     3-pass corner-distance contraction with gap-proportional space insertion,
+     `enforce_same_font`, ligature recomposition, loose-box geometry. Plus
+     docling's markdown escaping, wrap dehyphenation, paragraph-continuation
+     merging, and band-aware two-column reading order. Default; set
+     `DOCLING_LEGACY_LINES` for the old gap-heuristic join.
+   - Output is measured **byte-for-byte against live docling** (PDF_CONFORMANCE.md):
+     **4 / 14 exact**, the rest close. The remaining gaps are model-level
+     (TableFormer structure on complex tables, layout classification, RTL).
 
 6. **Extracted image bytes are real but not byte-identical.** Cropped/embedded
    pixels are correct, but the PNG re-encoding differs from docling's, so the
@@ -175,7 +191,10 @@ Explicitly **not done**, with the reason:
 - **VLM pipelines** (SmolDocling / remote VLM) and **enrichment models** (picture
   classification, formula understanding, code understanding). Model-bound; out of
   scope for the discriminative port.
-- **TableFormer.** Replaced by geometric table reconstruction (§4.5).
+
+(**TableFormer is now done** — ported to ONNX and run on every table region; see
+§5 and `PDF_CONFORMANCE.md`. Geometric reconstruction remains only as the fallback
+when the TableFormer graphs aren't present.)
 - **XML DocLang / DocTags** input backend — no `.dclg` sources in the corpus to
   verify against, and not in the requested scope.
 - **Older patent schemas.** USPTO covers the modern `v4x` XML only; the

PDF_CONFORMANCE.md

@@ -0,0 +1,83 @@
+# PDF conformance
+
+How close the Rust PDF pipeline gets to docling's **default** Markdown, measured
+byte-for-byte against the committed groundtruth (`tests/data/pdf/groundtruth/*.md`).
+The groundtruth is regenerated from **live published docling**, so it agrees with
+`scripts/conformance.sh pdf`.
+
+> Measure locally with `scripts/pdf_groundtruth.sh` (diffs the checked-in
+> reference; no docling install needed) or `scripts/conformance.sh pdf` (installs
+> docling and diffs against it). Diff = changed lines vs the groundtruth (one
+> changed line counts as 2).
+
+## Current state
+
+**4 / 14 groundtruth PDFs are byte-for-byte exact.**
+
+| PDF | diff | dominant remaining blocker |
+|---|---:|---|
+| picture_classification | **exact** | — |
+| code_and_formula | **exact** | — |
+| multi_page | **exact** | — |
+| 2305.03393v1-pg9 | **exact** | — (TableFormer table, cell-for-cell) |
+| right_to_left_01 | 2 | RTL justified double-space |
+| right_to_left_02 | 8 | RTL bidi spacing |
+| amt_handbook_sample | 14 | justified spacing + figure captions |
+| 2305.03393v1 | 40 | title-page reading order + author-ID run spacing |
+| right_to_left_03 | 66 | RTL bidi |
+| normal_4pages | 74 | reading order |
+| table_mislabeled_as_picture | 110 | layout over-detects tables (survey rendered as tables) |
+| 2206.01062 | 234 | TableFormer multi-row headers + title-page reading order |
+| 2203.01017v2 | 247 | TableFormer structure + reading order |
+| redp5110_sampled | 271 | TOC mis-classified as a picture; cover-page ordering |
+
+The close ones (`right_to_left_01/02`, `amt_handbook`) are 1–2 fixes from exact —
+the realistic path to 50% (7/14).
+
+## How the pipeline works
+
+pdfium extracts the glyph layer and renders each page to a bitmap; an ONNX stack
+(layout detection, TableFormer, PaddleOCR) interprets it; regions are assembled in
+reading order into a `DoclingDocument`. Tables use **TableFormer** (image encoder
++ autoregressive OTSL structure decoder + cell-bbox decoder, ported and exported
+to ONNX in `tableformer.rs`) on a cv2-exact preprocessed crop (`resample.rs`); the
+structure + matched cell text reproduce docling's padded GitHub tables (2305-pg9
+is cell-for-cell exact).
+
+### Text reconstruction: docling-parse line sanitizer (ported)
+
+The byte-exact ceiling used to be the **text extractor** — pdfium differs from
+docling's own `docling-parse` C++ parser at text-run boundaries. We closed it by
+porting docling-parse's line sanitizer (`src/parse/page_item_sanitators/cells.h`
+→ `dp_lines.rs`): a 3-pass corner-distance contraction (LTR → RTL → LTR-reverse)
+with `merge_with` space insertion (one space when the gap exceeds
+0.33×avg-char-width, plus literal space glyphs), `enforce_same_font`, ligature
+recomposition (same-loose-box glyphs become one cell), and loose-box geometry
+(uniform font ascent/descent), fed by pdfium glyph cells. It is the **default**
+(set `DOCLING_LEGACY_LINES` to fall back to the old gap heuristic). This fixed
+justified double-spacing, the space before `:`, lam-alef ordering, and fi/ffi
+ligatures — and got `multi_page` to byte-exact.
+
+Other text/serializer fixes matching docling: markdown escaping (`_`→`\_`, then
+HTML-escape `&`/`<`/`>`), U+2212→`-`, `@`-glue (`mAP @0.5`), wrap dehyphenation,
+paragraph-continuation merging across column/page breaks, and band-aware
+two-column reading order (full-width regions break the columns into bands).
+
+## Remaining blockers (model-level)
+
+These yield smaller or uncertain gains than the text-layer work already shipped.
+
+1. **TableFormer structure on complex tables.** Multi-row headers / spans on the
+   big papers (2206, 2203) differ from docling's OTSL prediction; one cell-
+   structure diff cascades through the padded columns into many row diffs
+   (2206's ~92 table-row diffs trace to ~4 structure diffs).
+2. **Layout classification.** The layout ONNX classifies redp5110's
+   table-of-contents as a *picture* (docling renders it as a table) and
+   table_mislabeled's survey as *tables* (docling renders lists/text) — opposite
+   classifications, not a text problem.
+3. **Complex title-page reading order.** Author-block / abstract interleaving on
+   the academic papers (band reading-order handles the full-width title; the
+   in-column author/abstract order is still off).
+4. **RTL justified double-spaces.** pdfium emits zero-width space boxes, losing
+   the literal space that, with the inserted one, forms a justified double
+   (`right_to_left_01`). Needs space-box reconstruction that estimates width.

README.md

@@ -166,6 +166,40 @@ The comparison scripts install the latest published Python `docling` from PyPI
 into `.venv-compare` automatically on first run. See
 [`COMPARING.md`](./COMPARING.md).
 
+## Performance
+
+`scripts/performance.sh` runs the **largest fixture of each supported type** through
+both engines (published Python `docling` vs the Rust release binary) and reports
+peak RSS, CPU utilization, and conversion time. Ratios below are docling ÷
+fleischwolf — bigger means Rust wins by more.
+
+| File | Size | Peak-memory ratio | CPU ratio | Warm-conversion speedup |
+|---|---:|---:|---:|---:|
+| `2203.01017v2.pdf` (PDF, 47 pp) | 6.9 MB | **2.2× less** | 1.3× | 1.2× |
+| `docx_rich_tables_01.docx` (DOCX) | 3.1 MB | **41× less** | 2.7× | 21× |
+| `wiki_duck.html` (HTML) | 240 KB | **57× less** | 3.2× | 46× |
+| `elife-56337.nxml` (JATS XML) | 180 KB | **61× less** | 2.9× | 10× |
+| `xlsx_04_inflated.xlsx` (XLSX) | 168 KB | **59× less** | 2.9× | 12× |
+| `powerpoint_with_image.pptx` (PPTX) | 80 KB | **57× less** | 2.8× | 4.4× |
+| `wiki.md` (Markdown) | 8 KB | **58× less** | 2.9× | 1.3× |
+| `csv-comma.csv` (CSV) | 4 KB | **66× less** | 2.9× | 0.6× † |
+
+- **Peak memory** is where Rust wins decisively: a declarative conversion holds a
+  few MB versus docling's ~750 MB (it imports torch even for non-ML formats). The
+  PDF runs the full ML pipeline in both engines (torch vs ONNX), so the gap there
+  is 2.2× rather than 50×+, but Rust still peaks at 1.4 GB vs docling's 3.1 GB.
+- **CPU**: docling spreads across 2.7–3.2 cores for declarative work that Rust does
+  on a single core (~100%); on the PDF both go multi-core (Rust 525% vs docling
+  674%).
+- **Warm-conversion speedup** isolates the parse/convert work — it times docling
+  *in-process* (excluding its ~3 s interpreter + import startup) against the Rust
+  whole-process figure. Rust wins on substantial inputs (HTML 46×, DOCX 21×); the
+  end-to-end figure, which re-pays docling's startup every invocation, is **377–
+  1190× faster** for the declarative formats.
+- † For trivial inputs (a 4 KB CSV) the conversion itself is microseconds, so Rust's
+  own process startup dominates its number while warm-Python excludes startup — the
+  warm metric understates Rust there. End-to-end, the CSV is **1190× faster** in Rust.
+
 ## Layout
 
 | Crate | Role | Python analogue |

crates/fleischwolf-core/src/document.rs

@@ -19,6 +19,11 @@ pub struct DoclingDocument {
     /// (the default) reproduces docling's legacy output byte-for-byte; `true`
     /// emits cleaner, more conformant Markdown. Set by `DocumentConverter`.
     pub strict_markdown: bool,
+    /// Emit tables in the compact `| a | b |` / `| - | - |` form rather than
+    /// docling-core's width-padded GitHub serializer. The PDF backend sets this
+    /// (its committed groundtruth corpus predates the padded serializer); DOCX/HTML
+    /// leave it `false` to match current published docling.
+    pub compact_tables: bool,
 }
 
 /// A single piece of document content.
@@ -92,6 +97,7 @@ impl DoclingDocument {
             name: name.into(),
             nodes: Vec::new(),
             strict_markdown: false,
+            compact_tables: false,
         }
     }