Skip to content

Commit 54031da

Browse files
artizclaude
artiz
and
claude
committed
docs(pdf): add PDF conformance roadmap + groundtruth measurement script
PDF_CONFORMANCE.md records the current per-PDF byte-conformance vs the docling groundtruth (1/14 exact after this PR) and scopes the remaining blockers: - Text-stream extraction — why raw chars / GetBoundedText both fail, and that the real fix needs an FPDFText_GetText per-char-range binding that pdfium-render 0.8.37 doesn't expose (upstream/FFI work). - TableFormer — weights, the autoregressive OTSL decoding loop, cell matching, span-aware serialization (six table PDFs blocked on it). - RTL/bidi, duplicate glyphs, code fencing. scripts/pdf_groundtruth.sh measures it (no docling install needed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 7d565a9 commit 54031da

2 files changed

Lines changed: 186 additions & 0 deletions

File tree

‎PDF_CONFORMANCE.md‎

Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
# PDF conformance roadmap
2+
3+
How close the Rust PDF pipeline gets to docling's **default** Markdown, measured
4+
byte-for-byte against the committed groundtruth (`tests/data/pdf/groundtruth/*.md`),
5+
and what it would take to close the remaining gap.
6+
7+
> Measure locally with `scripts/pdf_groundtruth.sh` (no docling install needed —
8+
> it diffs against the checked-in reference). The numbers below are the current
9+
> state.
10+
11+
## Current state
12+
13+
**1 / 14 groundtruth PDFs are byte-for-byte exact** (`picture_classification`);
14+
the rest are blocked on one of the categories below. Diff = changed lines vs the
15+
groundtruth (one changed line counts as 2).
16+
17+
| PDF | diff | dominant blocker |
18+
|---|---:|---|
19+
| picture_classification | **exact** | — |
20+
| right_to_left_01 | 4 | RTL/bidi |
21+
| code_and_formula | 6 | inter-run spacing + code fencing |
22+
| right_to_left_02 | 8 | RTL/bidi |
23+
| amt_handbook_sample | 12 | double-spaces, duplicate glyphs, fractions |
24+
| 2305.03393v1-pg9 | 25 | table structure |
25+
| right_to_left_03 | 74 | RTL/bidi |
26+
| multi_page | 76 | inter-run spacing + line-wrap hyphens |
27+
| normal_4pages | 108 | reading order (CJK) |
28+
| 2305.03393v1 | 152 | table structure |
29+
| table_mislabeled_as_picture | 151 | table structure |
30+
| 2203.01017v2 | 346 | table structure (+ inter-run spacing) |
31+
| 2206.01062 | 321 | table structure |
32+
| redp5110_sampled | 342 | table structure |
33+
34+
Shipped in this PR (no regressions; `pdf_conformance` stays 76/76):
35+
de-hyphenation + typography normalization, `<!-- formula-not-decoded -->`,
36+
caption-before-image pairing, and (strict-mode only) punctuation tightening.
37+
38+
Reaching ~50% exact requires the two big items below: **text-stream extraction**
39+
(unlocks the spacing-bound PDFs) and **TableFormer** (unlocks the six
40+
table-bound PDFs).
41+
42+
---
43+
44+
## Blocker 1 — inter-run text spacing (a.k.a. "text-stream extraction")
45+
46+
**Symptom.** pdfium splits a visual line into multiple style *segments* (a
47+
citation's superscripts, a code line's tokens, mixed fonts). We emit one cell
48+
per segment and join them with single spaces, so the real inter-run spacing is
49+
lost: `[ 37 , 36 ]` instead of `[37, 36]`, `function add ( a , b )` instead of
50+
`function add(a, b)`. docling reads text via pypdfium2's `get_text_range`
51+
(`FPDFText_GetText`), which inserts spaces from each glyph's *advance* and so
52+
reproduces the PDF's real spacing.
53+
54+
**What was tried in this PR and why each failed** (all reverted):
55+
56+
1. **Raw char API** (`PdfPageText::chars()` → `unicode_char()` + `loose_bounds()`,
57+
concatenated per line). pdfium's per-char list is *unreliable*: some lines
58+
come back with no space characters at all (`Thiscontentisextremelyvaluablefor`)
59+
and the char order is occasionally scrambled. Net regression.
60+
2. **`inside_rect()`** (`FPDFText_GetBoundedText`) over a whole line's bounding
61+
box. `GetBoundedText` ≠ `GetText`: it *drops* inter-run spaces on
62+
multi-segment lines (`{ahn,nli,mly,taa}@zurich` vs docling's
63+
`{ ahn,nli,mly,taa } @zurich`) and *bleeds* glyphs from vertically adjacent
64+
lines (`nevertheless exLines of different…`). Net regression.
65+
3. **Hybrid** (segment text for single-segment lines, `inside_rect` only for
66+
multi-segment lines). Same `GetBoundedText` divergence on exactly the lines
67+
that need fixing.
68+
69+
**Root cause / the real fix.** `segment.text()` is itself `inside_rect(segment.
70+
bounds())` — i.e. the *only* reliable text unit pdfium-render exposes is a single
71+
style run. What docling uses, `FPDFText_GetText(textpage, start_index, count, …)`
72+
for an arbitrary **character range**, is *not* wrapped by `pdfium-render`
73+
0.8.37. The path forward is to get that call:
74+
75+
- add a thin binding for `FPDFText_GetText` over a char range (upstream PR to
76+
`pdfium-render`, or call it through the crate's `PdfiumLibraryBindings` handle
77+
directly), then
78+
- group segments into lines (by vertical band, splitting at column gutters — the
79+
clustering already prototyped in this PR), map each line to its `[start, count]`
80+
char range, and read the whole line with `GetText`.
81+
82+
This is the single highest-leverage change for default-mode conformance: it
83+
fixes citations, inline code, fractions, and the justified-text double spaces,
84+
and unblocks `multi_page` and `code_and_formula` (the latter also needs code
85+
regions rendered as fenced blocks). **Stopgap shipped:** `--strict` tightens the
86+
citation/parenthetical spacing at serialization time, so strict Markdown already
87+
reads cleanly even though default mode still mirrors the segment spacing.
88+
89+
Also needed alongside it:
90+
- **Line-wrap de-hyphenation for real hyphens.** We already drop the U+0002 soft
91+
hyphen; `multi_page` wraps words with a real `-` (`professi-`/`onal`), which
92+
needs line-end-hyphen detection during the line join.
93+
- **Double-space preservation.** docling keeps the PDF's wide justified spacing
94+
(`the stainless steel nuts`); `clean_text` currently collapses runs of
95+
whitespace. With `GetText` per line, stop collapsing intra-line spacing.
96+
97+
## Blocker 2 — table structure (TableFormer)
98+
99+
**Symptom.** Six PDFs (`2206.01062`, `2305.03393v1[-pg9]`, `redp5110_sampled`,
100+
`table_mislabeled_as_picture`, and the table on `2203.01017v2`) are dominated by
101+
table differences. We reconstruct grids *geometrically* (cluster cells into
102+
rows/columns); docling runs **TableFormer**, an autoregressive transformer that
103+
predicts the table structure as an OTSL/HTML tag sequence plus per-cell bounding
104+
boxes, which recovers spanning headers and merged cells we cannot.
105+
106+
**Scope of a port** (large — own PR, likely staged over several):
107+
108+
1. **Weights.** TableFormer ships in `docling-ibm-models` (`TableModel04_rs`,
109+
"accurate"/"fast" variants). Export the encoder + the two decoders to ONNX
110+
from the published checkpoint; confirm the license permits redistribution of
111+
a converted model.
112+
2. **Inference loop.** Unlike the layout/OCR models (single `Session::run`),
113+
TableFormer is **autoregressive**: encode the table-crop image once, then step
114+
the structure decoder to emit OTSL tokens until `<end>`, feeding each token
115+
back in. The cell-bbox decoder runs per predicted cell. This is a real
116+
decoding loop in `fleischwolf-pdf`, not a one-shot call — budget for KV-cache
117+
handling and a token vocabulary/OTSL grammar.
118+
3. **Cell content.** Map predicted cell bounding boxes back onto the PDF text
119+
cells (we have these) to fill cell text — the same matching docling does for
120+
"PDF" tables (it does not OCR programmatic tables).
121+
4. **Serialization.** Convert the predicted OTSL grid (with row/col spans) to the
122+
`Table` node; the Markdown table serializer already exists but assumes a plain
123+
grid, so spans need representing.
124+
125+
A cheaper interim improvement (not docling-exact, but closes some diff): better
126+
geometric reconstruction — detect header rows, merge obvious spanning cells, and
127+
handle the multi-line header cells that currently shatter into many columns.
128+
129+
## Blocker 3 — RTL / bidi (Arabic)
130+
131+
`right_to_left_01/02/03`. Two compounding issues: (a) reading order — Latin runs
132+
embedded in RTL text and the overall right-to-left flow are emitted left-to-right
133+
(`Python و ة R` vs `R و Python`); we'd need Unicode bidi reordering of each line.
134+
(b) Arabic shaping — pdfium returns presentation-form / decomposed sequences that
135+
differ from docling's (`اإل` vs `الإ`), needing NFC-ish normalization of the
136+
Arabic block. Both are self-contained but specialized; lower priority than 1–2.
137+
138+
## Smaller items
139+
140+
- **Duplicate glyphs** (`amt_handbook`: `T he`, `F Figure 7-26 6`). pdfium emits
141+
doubled glyphs for some bold/overlapping text; needs de-duplication of
142+
overlapping cells.
143+
- **Code regions** → fenced ```` ``` ```` blocks with the caption *after* (code
144+
captions trail; figure captions lead). Pairs with Blocker 1 for the code text.

‎scripts/pdf_groundtruth.sh‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
#!/usr/bin/env bash
2+
#
3+
# Per-PDF byte-conformance of the Rust pipeline vs the committed docling
4+
# groundtruth (tests/data/pdf/groundtruth/*.md). Unlike conformance.sh this needs
5+
# no docling install — it diffs against the checked-in reference. Use it to track
6+
# how many groundtruth PDFs are byte-for-byte exact (see PDF_CONFORMANCE.md).
7+
#
8+
# Usage: scripts/pdf_groundtruth.sh
9+
10+
set -euo pipefail
11+
cd "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/.."
12+
13+
export PDFIUM_DYNAMIC_LIB_PATH="${PDFIUM_DYNAMIC_LIB_PATH:-$(pwd)/.pdfium/lib}"
14+
export DOCLING_LAYOUT_ONNX="${DOCLING_LAYOUT_ONNX:-$(pwd)/models/layout_heron.onnx}"
15+
export DOCLING_OCR_REC_ONNX="${DOCLING_OCR_REC_ONNX:-$(pwd)/models/ocr_rec.onnx}"
16+
export DOCLING_OCR_DICT="${DOCLING_OCR_DICT:-$(pwd)/models/ppocr_keys_v1.txt}"
17+
18+
cargo build --release --quiet -p fleischwolf-cli
19+
BIN=./target/release/fleischwolf
20+
21+
exact=0
22+
total=0
23+
printf "%-34s %12s\n" "PDF" "DIFF-LINES"
24+
printf "%-34s %12s\n" "---" "----------"
25+
for gt in tests/data/pdf/groundtruth/*.md; do
26+
stem="$(basename "$gt" .md)"
27+
src="tests/data/pdf/sources/$stem.pdf"
28+
[[ -f "$src" ]] || continue
29+
total=$((total + 1))
30+
out="$("$BIN" "$src" 2>/dev/null || echo '<ERROR>')"
31+
# Compare trailing-newline-insensitively; one changed line counts as 2.
32+
d="$(diff <(printf '%s' "$out") <(printf '%s' "$(cat "$gt")") | grep -cE '^[<>]' || true)"
33+
if [[ "$d" -eq 0 ]]; then
34+
exact=$((exact + 1))
35+
mark="EXACT"
36+
else
37+
mark="$d"
38+
fi
39+
printf "%-34s %12s\n" "$stem" "$mark"
40+
done
41+
echo
42+
echo "Fully conformant: $exact / $total"

0 commit comments

Comments
 (0)