fix(pdf): emit padded tables + align groundtruth to live docling

PDF tables were compact to match the stale committed groundtruth, but
conformance.sh (and DOCX/HTML) score against live published docling, which uses
padded GitHub tables — so the compact rows showed as cosmetic diffs (e.g.
2305-pg9 = 14 vs live). The PDF backend no longer opts into compact tables, so it
emits the padded format like every other backend.

Regenerated the committed PDF groundtruth from live docling (the conformance
reference) via scripts/docling_convert.py, so the fixtures and conformance.sh
agree. Result vs live docling: 2305-pg9 → EXACT (4/14), and the table papers drop
sharply (2203 321→275, 2206 304→276, 2305v1 94→68) now that the table format
matches. multi_page/code_and_formula/picture_classification stay exact.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: artiz

crates/fleischwolf-pdf/src/lib.rs

@@ -105,9 +105,6 @@ impl Pipeline {
         // is gigabytes for a multi-thousand-page document and drives the machine
         // into swap).
         let mut doc = DoclingDocument::new(name);
-        // The PDF groundtruth corpus predates docling-core's padded table
-        // serializer, so PDF output uses the compact `| a | b |` table form.
-        doc.compact_tables = true;
         pdfium_backend::for_each_page(bytes, password, |n, _total, mut page| {
             self.process_one_page(n, &mut page, &mut doc)
         })?;
@@ -187,7 +184,6 @@ impl Pipeline {
         name: &str,
     ) -> Result<DoclingDocument, PdfError> {
         let mut doc = DoclingDocument::new(name);
-        doc.compact_tables = true; // compact table form for PDF/image/METS output
         for (n, page) in pages.iter_mut().enumerate() {
             self.process_one_page(n, page, &mut doc)?;
         }

tests/data/pdf/groundtruth/2203.01017v2.md

@@ -14,17 +14,17 @@ The occurrence of tables in documents is ubiquitous. They often summarise quanti
 
 ## a. Picture of a table:
 
-| 1 |
-| - |
+| 1   |
+|-----|
 
 7
 
-| 0 | 1 2 1 | 1 2 1 | 1 2 1 |
-| - | - | - | - |
-| 3 4 | 5 | 6 | 7 |
-| 9 13 | 10 | 11 | 12 |
-| 8 2 | 14 | 15 | 16 |
-| 17 | 18 | 19 | 20 |
+| 0    |   1 2 1 |   1 2 1 |   1 2 1 |
+|------|---------|---------|---------|
+| 3 4  |       5 |       6 |       7 |
+| 9 13 |      10 |      11 |      12 |
+| 8 2  |      14 |      15 |      16 |
+| 17   |      18 |      19 |      20 |
 
 Figure 1: Picture of a table with subtle, complex features such as (1) multi-column headers, (2) cell with multi-row text and (3) cells with no content. Image from PubTabNet evaluation set, filename: 'PMC2944238 004 02'.
 
@@ -85,14 +85,14 @@ In this regard, we have prepared four synthetic datasets, each one containing 15
 
 Table 1: Both 'Combined-Tabnet' and 'CombinedTabnet' are variations of the following: (*) The CombinedTabnet dataset is the processed combination of PubTabNet and Fintabnet. (**) The combined dataset is the processed combination of PubTabNet, Fintabnet and TableBank.
 
-|  | Tags | Bbox | Size | Format |
-| - | - | - | - | - |
-| PubTabNet | 3 | 3 | 509k | PNG |
-| FinTabNet | 3 | 3 | 112k | PDF |
-| TableBank | 3 | 7 | 145k | JPEG |
-| Combined-Tabnet(*) | 3 | 3 | 400k | PNG |
-| Combined(**) | 3 | 3 | 500k | PNG |
-| SynthTabNet | 3 | 3 | 600k | PNG |
+|                    |   Tags |   Bbox | Size   | Format   |
+|--------------------|--------|--------|--------|----------|
+| PubTabNet          |      3 |      3 | 509k   | PNG      |
+| FinTabNet          |      3 |      3 | 112k   | PDF      |
+| TableBank          |      3 |      7 | 145k   | JPEG     |
+| Combined-Tabnet(*) |      3 |      3 | 400k   | PNG      |
+| Combined(**)       |      3 |      3 | 500k   | PNG      |
+| SynthTabNet        |      3 |      3 | 600k   | PNG      |
 
 Tab. 1 summarizes the various attributes of the datasets.
 
@@ -166,18 +166,18 @@ where T a and T b represent tables in tree structure HTML format. EditDist denot
 
 Structure. As shown in Tab. 2, TableFormer outperforms all SOTA methods across different datasets by a large margin for predicting the table structure from an image. All the more, our model outperforms pre-trained methods. During the evaluation we do not apply any table filtering. We also provide our baseline results on the SynthTabNet dataset. It has been observed that large tables (e.g. tables that occupy half of the page or more) yield poor predictions. We attribute this issue to the image resizing during the preprocessing step, that produces downsampled images with indistinguishable features. This problem can be addressed by treating such big tables with a separate model which accepts a large input image size.
 
-| Model | Dataset | Simple | TEDS Complex | All |
-| - | - | - | - | - |
-| EDD | PTN | 91.1 | 88.7 | 89.9 |
-| GTE | PTN | - | - | 93.01 |
-| TableFormer | PTN | 98.5 | 95.0 | 96.75 |
-| EDD | FTN | 88.4 | 92.08 | 90.6 |
-| GTE | FTN | - | - | 87.14 |
-| GTE (FT) | FTN | - | - | 91.02 |
-| TableFormer | FTN | 97.5 | 96.0 | 96.8 |
-| EDD | TB | 86.0 | - | 86.0 |
-| TableFormer | TB | 89.6 | - | 89.6 |
-| TableFormer | STN | 96.9 | 95.7 | 96.7 |
+| Model       | Dataset   | Simple   | TEDS Complex   |   All |
+|-------------|-----------|----------|----------------|-------|
+| EDD         | PTN       | 91.1     | 88.7           |  89.9 |
+| GTE         | PTN       | -        | -              | 93.01 |
+| TableFormer | PTN       | 98.5     | 95.0           | 96.75 |
+| EDD         | FTN       | 88.4     | 92.08          |  90.6 |
+| GTE         | FTN       | -        | -              | 87.14 |
+| GTE (FT)    | FTN       | -        | -              | 91.02 |
+| TableFormer | FTN       | 97.5     | 96.0           |  96.8 |
+| EDD         | TB        | 86.0     | -              |  86.0 |
+| TableFormer | TB        | 89.6     | -              |  89.6 |
+| TableFormer | STN       | 96.9     | 95.7           |  96.7 |
 
 Table 2: Structure results on PubTabNet (PTN), FinTabNet (FTN), TableBank (TB) and SynthTabNet (STN).
 
@@ -187,24 +187,24 @@ Cell Detection. Like any object detector, our Cell BBox Detector provides boundi
 
 Table 3: Cell Bounding Box detection results on PubTabNet, and FinTabNet. PP: Post-processing.
 
-| Model | Dataset | mAP | mAP (PP) |
-| - | - | - | - |
-| EDD+BBox | PubTabNet | 79.2 | 82.7 |
-| TableFormer | PubTabNet | 82.1 | 86.8 |
-| TableFormer | SynthTabNet | 87.7 | - |
+| Model       | Dataset     |   mAP | mAP (PP)   |
+|-------------|-------------|-------|------------|
+| EDD+BBox    | PubTabNet   |  79.2 | 82.7       |
+| TableFormer | PubTabNet   |  82.1 | 86.8       |
+| TableFormer | SynthTabNet |  87.7 | -          |
 
 Cell Content. In this section, we evaluate the entire pipeline of recovering a table with content. Here we put our approach to test by capitalizing on extracting content from the PDF cells rather than decoding from images. Tab. 4 shows the TEDs score of HTML code representing the structure of the table along with the content inserted in the data cell and compared with the ground-truth. Our method achieved a 5.3% increase over the state-of-the-art, and commercial solutions. We believe our scores would be higher if the HTML ground-truth matched the extracted PDF cell content. Unfortunately, there are small discrepancies such as spacings around words or special characters with various unicode representations.
 
 Table 4: Results of structure with content retrieved using cell detection on PubTabNet. In all cases the input is PDF documents with cropped tables.
 
-| Model | Simple | TEDS Complex | All |
-| - | - | - | - |
-| Tabula | 78.0 | 57.8 | 67.9 |
-| Traprange | 60.8 | 49.9 | 55.4 |
-| Camelot | 80.0 | 66.0 | 73.0 |
-| Acrobat Pro | 68.9 | 61.8 | 65.3 |
-| EDD | 91.2 | 85.4 | 88.3 |
-| TableFormer | 95.4 | 90.1 | 93.6 |
+| Model       |   Simple |   TEDS Complex |   All |
+|-------------|----------|----------------|-------|
+| Tabula      |     78.0 |           57.8 |  67.9 |
+| Traprange   |     60.8 |           49.9 |  55.4 |
+| Camelot     |     80.0 |           66.0 |  73.0 |
+| Acrobat Pro |     68.9 |           61.8 |  65.3 |
+| EDD         |     91.2 |           85.4 |  88.3 |
+| TableFormer |     95.4 |           90.1 |  93.6 |
 
 - a. Red - PDF cells, Green - predicted bounding boxes, Blue - post-processed predictions matched to PDF cells
 
@@ -349,24 +349,40 @@ Figure 9: Example of a table with big empty distance between cells.
 
 <!-- image -->
 
-Figure 10: Example of a complex table with empty cells.
+|           |         | ANOVA   | ANOVA   | 2            |
+|-----------|---------|---------|---------|--------------|
+| 3         | Sum Sq  | 5Df     | FValue  | Pr (>F)      |
+|           | 5745.2  |         | 266.75  | 4.64 × 10−9  |
+| 2onc      | 2191.39 |         | 0.7     | 2.76 × 10    |
+| R1× conc  | 2648.33 |         | 1.48    | 1.07 × 10 -6 |
+| Residuals | 236.91  |         | 6       |              |
 
-<!-- image -->
+Figure 10: Example of a complex table with empty cells.
 
 <!-- image -->
 
 Figure 11: Simple table with different style and empty cells.
 
+<!-- image -->
+
 Figure 13: Table predictions example on colorful table.
 
 <!-- image -->
 
+PDF Cells
+
 Figure 12: Simple table predictions and post processing.
 
 <!-- image -->
 
 Figure 14: Example with multi-line text.
 
+4 [4.0]
+
+9 μ4.1
+
+## PDF Cells
+
 Figure 15: Example with triangular table.
 
 <!-- image -->