Merge pull request #13 from artiz/claude/pdf-parser-code-path

feat(pdf): source code cells from the parser — fully retire pdfium's text path

Author: artiz

PDF_CONFORMANCE.md

@@ -87,9 +87,10 @@ on `lopdf`) that reconstructs each glyph's box from the *font's own advance
 widths* and the PDF text/graphics matrices — the same information docling-parse
 uses. It is the **default** text layer; set `DOCLING_PDFIUM_TEXT=1` to fall back
 to pdfium. Pages without a parseable text layer fall back to pdfium
-automatically, so scanned/OCR pages are unaffected. The parser also supplies the
-**word cells** TableFormer matches against (`DOCLING_PDFIUM_WORDS` keeps pdfium's);
-pdfium still provides page rasters, code cells, and link annotations.
+automatically, so scanned/OCR pages are unaffected. The parser supplies **all**
+text — prose, the **word cells** TableFormer matches against, and **code cells**
+(`DOCLING_PDFIUM_WORDS` reverts words+code to pdfium; `DOCLING_PDFIUM_TEXT`
+reverts everything). pdfium now does only page rasterisation + link annotations.
 
 The parser handles Type0/CID + Identity-H and simple Type1/TrueType fonts,
 ToUnicode CMaps (`bfchar`/`bfrange`), WinAnsi/MacRoman + `/Differences`
@@ -113,10 +114,13 @@ is a maximal run of glyphs the contraction merges *without* inserting a separato
 space, so the per-word segments split at exactly the `delta < gap` points — which
 reproduces docling-parse's `word_cells` byte-for-byte (377/377 on 2305-pg9). These
 are the per-word tokens TableFormer matches against table-grid cells, replacing
-pdfium's word cells (roadmap item 6). Code cells stay on pdfium for now — the
-parser's space-glyph-only code grouping drops monospace spacing
-(`function add` → `functionadd`); opt into the parser code path with
-`DOCLING_PARSER_CODE` once that's fixed.
+pdfium's word cells (roadmap item 6). **Code cells** come from the parser too,
+via a gap-based grouping (`Grouping::CodeGap`): the parser emits no space glyphs
+(a source space is a positioning gap), so a word breaks wherever the inter-glyph
+gap exceeds ~0.25× the line height, with no punctuation glue — `et al. 2000`
+keeps its space while `add(a,` / `b)` stay joined. `code_and_formula` is byte-exact
+(`function add(a, b) { return a + b; }`). With this, pdfium's text path is fully
+retired (rasters + links only).
 
 Other text/serializer/layout fixes matching docling: markdown escaping (`_`→`\_`,
 then HTML-escape `&`/`<`/`>`), typographic-punctuation normalization

PDF_PARSER_NOTES.md

@@ -163,9 +163,10 @@ numbering and footnote reading order.)
 4. ~~Korean quote normalization~~ — DONE (normal_4pages 74→54).
 5. **Now: 6/14 strict, 7/14 whitespace-normalized.** Blocker B (amt) needs a
    font-metrics layer for strict 7/14.
-6. **Word cells off pdfium — DONE (parser is now the default word source).** The
-   parser is already the sole *prose* source (item 2); **word** cells now come
-   from it too. The insight: docling-parse's `word_cells` are exactly the line
+6. **Drop pdfium's text path — DONE (parser is the sole text source).** The
+   parser was already the sole *prose* source (item 2); **word** and **code**
+   cells now come from it too, so pdfium does only rasterisation + links.
+   *Word cells:* the insight is that docling-parse's `word_cells` are exactly the line
    contraction's runs, split at the points where `merge_with` inserts a separator
    space (`delta < gap`). So instead of the legacy gap-heuristic
    (`words_from_glyphs`, which blob-joined TJ-spaced runs like
@@ -184,12 +185,25 @@ numbering and footnote reading order.)
    the now docling-faithful output (e.g. `bold ,` / `x 2`, which docling-parse's
    own `word_cells` confirm — the old `bold,` / `x2` were pdfium punct-gluing).
 
-   **Code cells still pdfium.** The parser's space-glyph-only code grouping drops
-   the inter-token spaces pdfium recovers in monospace listings
-   (`function add` → `functionadd`, a regression vs groundtruth), so code cells
-   stay on pdfium, gated behind `DOCLING_PARSER_CODE` for a future fix (the parser
-   needs faithful monospace spacing before the *code* path can retire too). pdfium
-   thus remains for code cells + page rasterisation + link annotations.
+   **Code cells off pdfium too — DONE (pdfium's text path fully retired).** The
+   first attempt at parser code cells used the space-glyph-only grouping, which
+   dropped the inter-token spaces pdfium recovers (`function add` → `functionadd`)
+   because the parser emits no space glyphs — a source space is a positioning
+   *gap*. The fix is a third grouping mode, `Grouping::CodeGap`: split on the
+   inter-glyph gap (a space wherever it exceeds ~0.25× the line height) but with
+   **no punctuation glue**, so a real gap always splits (`et al. 2000`, not
+   `et al.2000` — the prose glue rule is wrong for code) while genuinely touching
+   tokens stay joined (`add(a,` / `b)`). The parser's clean advance boxes make the
+   gap reliable here, where pdfium's overhanging loose boxes would over-split
+   (`f un c t i o n`) — which is why pdfium keeps the space-glyph path.
+   `code_and_formula` is byte-exact (`function add(a, b) { return a + b; }
+   console.log(add(3, 5));`); it's the default (opt out with `DOCLING_PDFIUM_WORDS`
+   or `DOCLING_PDFIUM_TEXT`). Two snapshots (`2305` llncsdoc, `redp5110`) drift in
+   garbled multi-column LaTeX/SQL "code" regions — conformance-**neutral** vs the
+   groundtruth (337/337, 175/175), regenerated to the parser output.
+
+   pdfium now does **only** page rasterisation + link annotations; all text
+   (prose, words, code) comes from the pure-Rust parser.
 
 ## Tooling (under `scripts/`)
 

crates/fleischwolf-pdf/src/pdfium_backend.rs

@@ -86,13 +86,15 @@ pub(crate) fn use_parser_words() -> bool {
     std::env::var("DOCLING_PDFIUM_WORDS").is_err() && std::env::var("DOCLING_PDFIUM_TEXT").is_err()
 }
 
-/// Whether to source **code** cells from the parser too. Off by default: the
-/// parser's space-glyph-only code grouping drops the inter-token spaces pdfium
-/// recovers in monospace listings (`function add` → `functionadd`), a regression
-/// vs the docling groundtruth. Opt in with `DOCLING_PARSER_CODE` once the parser
-/// emits faithful monospace spacing. `DOCLING_PDFIUM_TEXT` always wins (pdfium).
+/// Whether to source **code** cells from the parser too (the default) — the last
+/// text layer to leave pdfium, fully retiring its text path. The parser's
+/// gap-based code grouping ([`code_cells_from_glyphs`]) reconstructs monospace
+/// spacing from positioning gaps (`function add(a, b) { … }`), so it no longer
+/// drops the inter-token spaces the old space-glyph-only grouping lost
+/// (`functionadd`). Reverts to pdfium with `DOCLING_PDFIUM_WORDS` (alongside word
+/// cells) or `DOCLING_PDFIUM_TEXT` (all text).
 pub(crate) fn use_parser_code() -> bool {
-    std::env::var("DOCLING_PARSER_CODE").is_ok() && std::env::var("DOCLING_PDFIUM_TEXT").is_err()
+    std::env::var("DOCLING_PDFIUM_WORDS").is_err() && std::env::var("DOCLING_PDFIUM_TEXT").is_err()
 }
 
 fn bind() -> Result<Pdfium, PdfiumError> {
@@ -355,11 +357,11 @@ impl<'a> FfiText<'a> {
             let prose = if dp {
                 crate::dp_lines::line_cells(&g, page_h, false)
             } else {
-                lines_from_glyphs(&g, page_h, false)
+                lines_from_glyphs(&g, page_h, Grouping::Prose)
             };
             (
                 prose,
-                lines_from_glyphs(&g, page_h, true),
+                lines_from_glyphs(&g, page_h, Grouping::CodeSpaceOnly),
                 words_from_glyphs(&g, page_h),
             )
         };
@@ -548,18 +550,33 @@ fn glyphs(b: &dyn PdfiumLibraryBindings, tp: FPDF_TEXTPAGE, fetch_font: bool) ->
     out
 }
 
+/// How [`lines_from_glyphs`] splits a line into words.
+#[derive(Clone, Copy, PartialEq)]
+enum Grouping {
+    /// Gap heuristic + punctuation glue (`engines,`, `[37`, `98.5`) — prose.
+    Prose,
+    /// Split only at literal space glyphs, never glue — pdfium code cells.
+    /// pdfium's monospace listings carry a real space glyph at every source space,
+    /// and its overhanging loose boxes would make the gap heuristic over-split
+    /// (`f un c t i o n`), so honouring just the spaces reproduces the spacing.
+    CodeSpaceOnly,
+    /// Split on the inter-glyph **gap** (or a space glyph), but never glue — for
+    /// the parser's code cells: the parser emits no space glyphs (a source space
+    /// is a positioning gap), and its clean advance boxes make the gap reliable.
+    /// Unlike [`Grouping::Prose`] there is no punctuation glue, so a real gap
+    /// always splits (`et al. 2000`, not `et al.2000`) while genuinely touching
+    /// tokens stay joined (`add(a,` / `b)`).
+    CodeGap,
+}
+
 /// Group glyphs (document order) into words then lines, the way docling-parse
 /// does: a new **word** starts where the horizontal gap to the previous glyph
 /// exceeds ~0.2 × the font height (a real space is ~0.3 × height; letter
 /// tracking is smaller, so titles don't shatter); a new **line** starts where
 /// the baseline drops by ~half the font height (a superscript rises without
 /// dropping, so it stays on its line). Coordinates are flipped to top-left.
-/// `code` mode splits words **only** at pdfium's own space glyphs and never glues
-/// punctuation — monospace code has wide inter-glyph advances that the prose
-/// gap heuristic mistakes for spaces (`f un c t i o n`), but pdfium emits a real
-/// space glyph at every true gap, so honoring just those reproduces the source
-/// spacing (`function add(a, b)`).
-fn lines_from_glyphs(gs: &[Glyph], page_h: f32, code: bool) -> Vec<TextCell> {
+/// See [`Grouping`] for how each mode decides word boundaries.
+fn lines_from_glyphs(gs: &[Glyph], page_h: f32, mode: Grouping) -> Vec<TextCell> {
     let mut cells: Vec<TextCell> = Vec::new();
     let mut words: Vec<String> = Vec::new(); // words on the current line
     let mut word = String::new();
@@ -613,8 +630,11 @@ fn lines_from_glyphs(gs: &[Glyph], page_h: f32, code: bool) -> Vec<TextCell> {
                     && !pending_space
                     && (g.ch.is_ascii_digit() || g.ch.is_ascii_lowercase()));
             let word_gap = line_h.max(h) * 0.25;
-            new_word = if code {
+            new_word = if mode == Grouping::CodeSpaceOnly {
                 new_line || pending_space
+            } else if mode == Grouping::CodeGap {
+                // Gap-based, no glue: a real gap always splits, touching tokens join.
+                new_line || pending_space || g.l - p.r > word_gap
             } else if is_arabic(g.ch) || is_arabic(p.ch) {
                 // RTL runs right-to-left, so the inter-word gap is `p.l - g.r`. A
                 // real word space has a gap; pdfium also emits spurious zero-gap
@@ -652,11 +672,16 @@ fn lines_from_glyphs(gs: &[Glyph], page_h: f32, code: bool) -> Vec<TextCell> {
     cells
 }
 
-/// Code line cells from a glyph stream (parser or pdfium): split only at space
-/// glyphs so monospace runs keep their source spacing. Thin wrapper over
-/// [`lines_from_glyphs`] with `code = true`, for the parser text path.
+/// Code line cells from the **parser**'s glyph stream. Unlike pdfium — whose
+/// monospace listings carry explicit space glyphs (so [`Grouping::CodeSpaceOnly`]
+/// keeps their spacing) — the parser emits no space glyphs: a source space is a
+/// positioning gap. So code cells use [`Grouping::CodeGap`], which splits on the
+/// inter-glyph gap (a space wherever it exceeds ~0.25× the line height) but never
+/// glues punctuation, so `et al. 2000` keeps its space while `add(a,` / `b)` stay
+/// joined. The parser's clean advance boxes make the gap heuristic reliable here,
+/// where pdfium's overhanging loose boxes would over-split (`f un c t i o n`).
 pub(crate) fn code_cells_from_glyphs(gs: &[Glyph], page_h: f32) -> Vec<TextCell> {
-    lines_from_glyphs(gs, page_h, true)
+    lines_from_glyphs(gs, page_h, Grouping::CodeGap)
 }
 
 /// Per-word cells (each word's text + top-left bbox), using the same word/line

tests/snapshots/latex/sources/2305.03393/llncsdoc.pdf.md

@@ -113,7 +113,7 @@ Multiple affiliations are separated by \and , which automatically assures correc
 \and
 
 ```
-\institute{<name of an institute> \and <name of the next institute> \and <name of the next institute>} \email Inside \institute you can use \email{<email address>} \url and \url{<url>}
+\institute{<name of an institute> \and <name of the next institute> \and <name of the next institute>} Inside \institute you can use\email \email{<email address>} and\url \url{<url>}
 ```
 
 ```
@@ -143,7 +143,7 @@ Please note that, if email addresses are given in your paper, they will also be
 abstract ( env. ) The abstract is coded as follows:
 
 ```
-abstract(env.) The abstract is coded as follows: \begin{abstract} <Text of the summary of your paper> \end{abstract}
+The abstract is coded as follows:abstract(env.) \begin{abstract} <Text of the summary of your paper> \end{abstract}
 ```
 
 ```
@@ -167,7 +167,7 @@ From a technical point of view, the llncs document class does not require any sp
 The llncs document class supports some additional special characters:
 
 ```
-\grole yields > < \getsto yields ← → \lid yields < = \gid yields > =
+\grole yields > \lid yields < < \getsto yields ←→ = \gid yields > =
 ```
 
 If you need blackboard bold characters, i.e. for sets of numbers, please load the related A M S -T E Xfonts. If for some reason this is not possible you can also use the following commands from the llncs class:
@@ -200,7 +200,7 @@ proposition ( env. )
 example ( env. )
 
 ```
-property(env.) question(env.) exercise(env.) solution(env.)heading is bold as well: problem(env.) note(env.) \begin{case} <text> \end{case} \begin{conjecture} <text> \end{conjecture} \begin{example} <text> \end{example} \begin{exercise} <text> \end{exercise} \begin{note} <text> \end{note} \begin{problem} <text> \end{problem} remark(env.) \begin{property} <text> \end{property} \begin{question} <text> \end{question} \begin{remark} <text> \end{remark} \begin{solution} <text> \end{solution}
+exercise(env.) property(env.) question(env.) solution(env.) heading is bold as well: problem(env.) note(env.) \begin{case} <text> \end{case} \begin{conjecture} <text> \end{conjecture} \begin{example} <text> \end{example} \begin{exercise} <text> \end{exercise} \begin{note} <text> \end{note} \begin{problem} <text> \end{problem} remark(env.) \begin{property} <text> \end{property} \begin{question} <text> \end{question} \begin{remark} <text> \end{remark} \begin{solution} <text> \end{solution}
 ```
 
 claim ( env. ) Finally, there are also two unnumbered environments that have the run-in headproof ( env. ) ing in italics and the text in upright roman.
@@ -260,7 +260,7 @@ citeauthoryear
 Please note that this option does not automatically change your citations to the author/year style. It basically redefines the \bibitem command to take the publication year as an optional parameter that is displayed instead of an arabic number. Author name(s) and, if necessary, parentheses are to be typed manually. If your reference reads
 
 ```
-\bibitem[2016]{vdaalst:2016} van der Aalst, W.: Process Mining, 2nd ed. Springer, Heidelberg(2016) and is cited as follows:... is shown by van der Aalst(\cite{vdaalst:2016}) the resulting text will be: "... is shown by van der Aalst(2016)."
+\bibitem[2016]{vdaalst:2016} van der Aalst, W.: Process Mining, 2nd ed. Springer, Heidelberg(2016) and is cited as follows:... is shown by van der Aalst(\cite{vdaalst:2016}) the resulting text will be: "...is shown by van der Aalst(2016)."
 ```
 
 We encourage you to use Bib T E X for typesetting your references. For formatting the bibliography according to Springer's standard (for mathematics, physical sciences, and computer science), please use the bibliography style file splncs04.bst that comes with the llncs document class. You simply need to add \bibliographystyle{splncs04} to your document. DOIs should be provided in the doi field of your .bib database. Bib T E X will then automatically add them to your references. Please note that we do not provide an option to implement splncs04.bst

tests/snapshots/pdf/sources/redp5110_sampled.pdf.md

@@ -336,7 +336,7 @@ Figure 4-69   Index advice with no RCAC
 <!-- image -->
 
 ```
-WHEN QSYS2. VERIFY_GROUP_FOR_USER( SESSION_USER, 'TELLER') = 1 WHEN QSYS2. VERIFY_GROUP_FOR_USER( SESSION_USER, 'CUSTOMER') = 1 WHEN QSYS2. VERIFY_GROUP_FOR_USER( SESSION_USER, 'ADMIN') = 1 WHEN QSYS2. VERIFY_GROUP_FOR_USER( SESSION_USER, 'TELLER') = 1 WHEN QSYS2. VERIFY_GROUP_FOR_USER( SESSION_USER, 'CUSTOMER') = 1 124 Row and Column Access Control Support in IBM DB2 for iTHEN C. CUSTOMER_TAX_ID THEN( 'XXX-XX-' CONCAT QSYS2. SUBSTR( C. CUSTOMER_TAX_ID, 8, 4)) THEN C. CUSTOMER_TAX_ID THEN C. CUSTOMER_DRIVERS_LICENSE_NUMBER THEN C. CUSTOMER_DRIVERS_LICENSE_NUMBER THEN C. CUSTOMER_DRIVERS_LICENSE_NUMBER CREATE MASK BANK_SCHEMA.MASK_DRIVERS_LICENSE_ON_CUSTOMERS ON BANK_SCHEMA.CUSTOMERS AS C CREATE MASK BANK_SCHEMA.MASK_LOGIN_ID_ON_CUSTOMERS ON BANK_SCHEMA.CUSTOMERS AS C ELSE 'XXX-XX-XXXX' END ELSE '*************' END RETURN CASE RETURN CASE ENABLE; FOR COLUMN CUSTOMER_DRIVERS_LICENSE_NUMBER ENABLE; FOR COLUMN CUSTOMER_LOGIN_ID ALTER TABLE BANK_SCHEMA.CUSTOMERS WHEN QSYS2. VERIFY_GROUP_FOR_USER( SESSION_USER, 'ADMIN') = 1 WHEN QSYS2. VERIFY_GROUP_FOR_USER( SESSION_USER, 'CUSTOMER') = 1 WHEN QSYS2. VERIFY_GROUP_FOR_USER( SESSION_USER, 'ADMIN') = 1 WHEN QSYS2. VERIFY_GROUP_FOR_USER( SESSION_USER, 'CUSTOMER') = 1 WHEN QSYS2. VERIFY_GROUP_FOR_USER( SESSION_USER, 'ADMIN') = 1 WHEN QSYS2. VERIFY_GROUP_FOR_USER( SESSION_USER, 'CUSTOMER') = 1 THEN C. CUSTOMER_LOGIN_ID THEN C. CUSTOMER_LOGIN_ID THEN C. CUSTOMER_SECURITY_QUESTION THEN C. CUSTOMER_SECURITY_QUESTION THEN C. CUSTOMER_SECURITY_QUESTION_ANSWER THEN C. CUSTOMER_SECURITY_QUESTION_ANSWER CREATE MASK BANK_SCHEMA.MASK_SECURITY_QUESTION_ON_CUSTOMERS ON BANK_SCHEMA.CUSTOMERS AS C CREATE MASK BANK_SCHEMA.MASK_SECURITY_QUESTION_ANSWER_ON_CUSTOMERS ON BANK_SCHEMA.CUSTOMERS AS C ELSE '*****' END ELSE '*****' END ELSE '*****' END ACTIVATE ROW ACCESS CONTROL ACTIVATE COLUMN ACCESS CONTROL; RETURN CASE RETURN CASE ENABLE; FOR COLUMN CUSTOMER_SECURITY_QUESTION ENABLE; FOR COLUMN CUSTOMER_SECURITY_QUESTION_ANSWER ENABLE;
+124 Row and Column Access Control Support in IBM DB2 for iTHEN C. CUSTOMER_TAX_ID WHEN QSYS2. VERIFY_GROUP_FOR_USER( SESSION_USER, 'TELLER') = 1 THEN( 'XXX-XX-' CONCAT QSYS2. SUBSTR( C. CUSTOMER_TAX_ID, 8, 4)) WHEN QSYS2. VERIFY_GROUP_FOR_USER( SESSION_USER, 'CUSTOMER') = 1 THEN C. CUSTOMER_TAX_ID ELSE 'XXX-XX-XXXX' END ENABLE; CREATE MASK BANK_SCHEMA.MASK_DRIVERS_LICENSE_ON_CUSTOMERS ON BANK_SCHEMA.CUSTOMERS AS C WHEN QSYS2. VERIFY_GROUP_FOR_USER( SESSION_USER, 'ADMIN') = 1 THEN C. CUSTOMER_DRIVERS_LICENSE_NUMBER WHEN QSYS2. VERIFY_GROUP_FOR_USER( SESSION_USER, 'TELLER') = 1 THEN C. CUSTOMER_DRIVERS_LICENSE_NUMBER WHEN QSYS2. VERIFY_GROUP_FOR_USER( SESSION_USER, 'CUSTOMER') = 1 THEN C. CUSTOMER_DRIVERS_LICENSE_NUMBER ELSE '*************' END ENABLE; CREATE MASK BANK_SCHEMA.MASK_LOGIN_ID_ON_CUSTOMERS ON BANK_SCHEMA.CUSTOMERS AS C FOR COLUMN CUSTOMER_DRIVERS_LICENSE_NUMBER RETURN CASE FOR COLUMN CUSTOMER_LOGIN_ID RETURN CASE WHEN QSYS2. VERIFY_GROUP_FOR_USER( SESSION_USER, 'ADMIN') = 1 THEN C. CUSTOMER_LOGIN_ID WHEN QSYS2. VERIFY_GROUP_FOR_USER( SESSION_USER, 'CUSTOMER') = 1 THEN C. CUSTOMER_LOGIN_ID ELSE '*****' END ENABLE; CREATE MASK BANK_SCHEMA.MASK_SECURITY_QUESTION_ON_CUSTOMERS ON BANK_SCHEMA.CUSTOMERS AS C WHEN QSYS2. VERIFY_GROUP_FOR_USER( SESSION_USER, 'ADMIN') = 1 THEN C. CUSTOMER_SECURITY_QUESTION WHEN QSYS2. VERIFY_GROUP_FOR_USER( SESSION_USER, 'CUSTOMER') = 1 THEN C. CUSTOMER_SECURITY_QUESTION ELSE '*****' END ENABLE; CREATE MASK BANK_SCHEMA.MASK_SECURITY_QUESTION_ANSWER_ON_CUSTOMERS ON BANK_SCHEMA.CUSTOMERS AS C WHEN QSYS2. VERIFY_GROUP_FOR_USER( SESSION_USER, 'ADMIN') = 1 THEN C. CUSTOMER_SECURITY_QUESTION_ANSWER WHEN QSYS2. VERIFY_GROUP_FOR_USER( SESSION_USER, 'CUSTOMER') = 1 THEN C. CUSTOMER_SECURITY_QUESTION_ANSWER ELSE '*****' END ENABLE; ALTER TABLE BANK_SCHEMA.CUSTOMERS FOR COLUMN CUSTOMER_SECURITY_QUESTION RETURN CASE FOR COLUMN CUSTOMER_SECURITY_QUESTION_ANSWER RETURN CASE ACTIVATE ROW ACCESS CONTROL ACTIVATE COLUMN ACCESS CONTROL;
 ```
 
 <!-- image -->