pdf: demote uncaptioned text-panel pictures into paragraphs (#157)

A terms-and-conditions box exported as an image — the XXXLutz blue
"Folgen des Widerrufs" panel — survives every pipeline as a picture: its
words are not in the digital text layer, region-scoped OCR skips picture
interiors, and docling itself never serializes cells assigned to a picture
cluster. The panel's C.7–C.12 were simply gone from the markdown, in
normal, force-OCR and browser conversions alike.

Three shared changes:

- assemble::recover_text_panels: an *uncaptioned* `picture` whose cells
  (digital or OCR) form ≥ 3 lines, cover ≥ 20 % of its area and whose
  median line spans most of its width is a text panel, not a figure —
  replace it with per-paragraph `text` regions (paragraphs re-derived from
  the line pitch). A captioned picture is a genuine figure whatever it
  contains (the corpus' "Figure 3:" document screenshots are exactly as
  dense), and sparse-label charts fail the width/coverage gate — both keep
  their crops. Wired into every path: native finish_page, the shared
  refine_regions chain (browser digital), and the browser scanned/digital
  bare-picture OCR blocks.
- Native pipeline now OCRs big text-less picture crops (docling's
  bitmap_area_threshold: bitmap areas ≥ 5 % of the page, every page kind)
  the way the browser paths already did — the demotion's cell source on
  digital pages whose panels are embedded rasters. On OCR'd pages,
  non-demoting in-picture text is emitted beside the kept crop (browser
  scanned parity); on digital pages the speculative cells are discarded so
  docling-groundtruth photos stay silent and picture_classification stays
  byte-exact.
- ocr_prep::segment_lines: mask thin full-height "rule" columns (a panel's
  frame) out of the row-ink profile — the border bridged every inter-line
  gap and the whole framed panel OCR'd as one unreadable strip. The mask
  backs off when "rules" are a noticeable share of the width (that shape is
  an inverted dark-mode page, polarity normalization's job).

Also: force_full_page_ocr's cell-clearing moves from process() into
finish_page so the *batched* layout path honors the flag — it was silently
a no-op for multi-page documents (batch is the default).

Conformance: 5/14 strict + 6/14 ws-normalized holds. 2203 130→132,
normal_4pages 44→50, redp5110 194→196 — each a few lines of *recovered*
panel text docling drops (normal_4pages' cover publisher block), documented
in docs/PDF_CONFORMANCE.md. Everything else byte-identical; full battery
(docling-core/docling/asr/serve/pdf + wasm host tests) green, clippy clean.

Refs #157

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT

Author: artiz

crates/docling-pdf/src/assemble.rs

@@ -358,6 +358,143 @@ pub fn add_orphan_regions(regions: &mut Vec<Region>, cells: &[TextCell]) {
     regions.extend(merged);
 }
 
+/// Demote a `picture` region that is really a **text panel** — a paragraph block
+/// the layout model boxed as a figure because it is typeset on a colored
+/// background (terms-and-conditions callouts, quote boxes) — into ordinary
+/// `text` regions, one per paragraph, so its words are read instead of shipped
+/// as pixels. docling loses this text the same way (cells assigned to a picture
+/// cluster are never serialized); this is a deliberate improvement, not parity.
+///
+/// The gate is conservative so a genuine figure keeps its crop: the region must
+/// contain at least three text lines whose median width spans most of the panel
+/// (axis labels and chat bubbles are narrow and varied) and whose cells cover a
+/// substantial fraction of its area (a photo or chart with sparse labels does
+/// not). Paragraph boundaries are re-derived from the line pitch: a vertical gap
+/// clearly larger than the panel's own leading starts a new `text` region, so
+/// the panel doesn't collapse into one giant paragraph.
+///
+/// Works on any cell source — the digital text layer or OCR lines recognized
+/// from the picture crop — so the native and browser paths, with or without
+/// force-OCR, demote identically.
+pub fn recover_text_panels(regions: &mut Vec<Region>, cells: &[TextCell]) {
+    // A *captioned* picture is a genuine figure whatever it contains — the
+    // corpus is full of document screenshots ("Figure 3: …" above a page
+    // image) that are exactly as dense and wide as a text panel. Only an
+    // uncaptioned picture is a demotion candidate.
+    let captioned: Vec<bool> = regions
+        .iter()
+        .map(|r| {
+            r.label == "picture"
+                && regions.iter().any(|c| {
+                    c.label == "caption" && c.r.min(r.r) - c.l.max(r.l) > 0.0 && {
+                        let gap = if c.t >= r.b {
+                            c.t - r.b
+                        } else if r.t >= c.b {
+                            r.t - c.b
+                        } else {
+                            f32::MAX // vertically overlapping: not a caption
+                        };
+                        gap <= 25.0
+                    }
+                })
+        })
+        .collect();
+    let mut out: Vec<Region> = Vec::with_capacity(regions.len());
+    for (i, r) in regions.drain(..).enumerate() {
+        if r.label != "picture" || captioned[i] {
+            out.push(r);
+            continue;
+        }
+        let inside: Vec<&TextCell> = cells
+            .iter()
+            .filter(|c| {
+                !c.text.trim().is_empty() && {
+                    let ca = area(c.l, c.t, c.r, c.b).max(1.0);
+                    inter(&r, c.l, c.t, c.r, c.b) / ca > 0.5
+                }
+            })
+            .collect();
+        // Group the contained cells into lines by vertical overlap (the same
+        // rule region_text orders by), tracking each line's union box.
+        let mut lines: Vec<(f32, f32, f32, f32)> = Vec::new(); // (t, b, l, r)
+        for c in &inside {
+            let (ct, cb) = (c.t.min(c.b), c.t.max(c.b));
+            match lines.iter_mut().find(|(lt, lb, _, _)| {
+                let ov = cb.min(*lb) - ct.max(*lt);
+                ov > 0.5 * (cb - ct).min(*lb - *lt).max(1.0)
+            }) {
+                Some((lt, lb, ll, lr)) => {
+                    *lt = lt.min(ct);
+                    *lb = lb.max(cb);
+                    *ll = ll.min(c.l);
+                    *lr = lr.max(c.r);
+                }
+                None => lines.push((ct, cb, c.l, c.r)),
+            }
+        }
+        let panel_w = (r.r - r.l).max(1.0);
+        let coverage = inside.iter().map(|c| area(c.l, c.t, c.r, c.b)).sum::<f32>()
+            / area(r.l, r.t, r.r, r.b).max(1.0);
+        let mut widths: Vec<f32> = lines.iter().map(|(_, _, l, rr)| rr - l).collect();
+        widths.sort_by(f32::total_cmp);
+        let text_panel =
+            lines.len() >= 3 && coverage >= 0.2 && widths[widths.len() / 2] >= 0.45 * panel_w;
+        if !text_panel {
+            out.push(r);
+            continue;
+        }
+        lines.sort_by(|a, b| a.0.total_cmp(&b.0));
+        let mut heights: Vec<f32> = lines.iter().map(|(t, b, _, _)| b - t).collect();
+        heights.sort_by(f32::total_cmp);
+        let h = heights[heights.len() / 2].max(1.0);
+        let mut gaps: Vec<f32> = lines
+            .windows(2)
+            .map(|w| (w[1].0 - w[0].1).max(0.0))
+            .collect();
+        gaps.sort_by(f32::total_cmp);
+        let leading = if gaps.is_empty() {
+            0.0
+        } else {
+            gaps[gaps.len() / 2]
+        };
+        let brk = (1.8 * leading).max(0.75 * h);
+        let mut para: Option<(f32, f32, f32, f32)> = None; // (l, t, r, b) union
+        for (t, b, l, rr) in &lines {
+            match &mut para {
+                Some((pl, _, pr, pb)) if *t - *pb <= brk => {
+                    *pl = pl.min(*l);
+                    *pr = pr.max(*rr);
+                    *pb = pb.max(*b);
+                }
+                _ => {
+                    if let Some((pl, pt, pr, pb)) = para.take() {
+                        out.push(Region {
+                            label: "text",
+                            score: r.score,
+                            l: pl,
+                            t: pt,
+                            r: pr,
+                            b: pb,
+                        });
+                    }
+                    para = Some((*l, *t, *rr, *b));
+                }
+            }
+        }
+        if let Some((pl, pt, pr, pb)) = para {
+            out.push(Region {
+                label: "text",
+                score: r.score,
+                l: pl,
+                t: pt,
+                r: pr,
+                b: pb,
+            });
+        }
+    }
+    *regions = out;
+}
+
 /// Drop a `picture` detection that is a small, empty, low-confidence margin box on
 /// a **text page** — a false positive the RT-DETR layout sometimes emits (e.g.
 /// `right_to_left_02`'s phantom right-column picture, score 0.40); docling does not
@@ -1743,6 +1880,88 @@ mod tests {
     use crate::pdfium_backend::{LinkAnnot, PdfPage, TextCell};
     use docling_core::Node;
 
+    /// A colored terms-and-conditions panel detected as `picture` demotes into
+    /// per-paragraph `text` regions (the blank line between C.7 and C.8 splits
+    /// them); a chart whose only text is a few narrow axis labels keeps its
+    /// crop untouched.
+    #[test]
+    fn text_panels_demote_to_paragraphs_but_charts_keep_their_crop() {
+        let cell = |text: &str, l: f32, t: f32, r: f32, b: f32| TextCell {
+            text: text.to_string(),
+            l,
+            t,
+            r,
+            b,
+        };
+        let panel = Region {
+            label: "picture",
+            score: 0.9,
+            l: 0.0,
+            t: 0.0,
+            r: 100.0,
+            b: 100.0,
+        };
+        // Three tight lines, a blank-line gap, two more: two paragraphs.
+        let cells = vec![
+            cell(
+                "C.7. Wenn Sie diesen Vertrag widerrufen,",
+                5.0,
+                10.0,
+                95.0,
+                18.0,
+            ),
+            cell(
+                "haben wir Ihnen alle Zahlungen, die wir",
+                5.0,
+                20.0,
+                95.0,
+                28.0,
+            ),
+            cell(
+                "von Ihnen erhalten haben, zurückzuzahlen.",
+                5.0,
+                30.0,
+                90.0,
+                38.0,
+            ),
+            cell(
+                "C.8. Wir können die Rückzahlung verweigern,",
+                5.0,
+                52.0,
+                95.0,
+                60.0,
+            ),
+            cell(
+                "bis wir die Waren wieder zurückerhalten haben.",
+                5.0,
+                62.0,
+                92.0,
+                70.0,
+            ),
+        ];
+        let mut regions = vec![panel.clone()];
+        super::recover_text_panels(&mut regions, &cells);
+        assert_eq!(
+            regions.iter().map(|r| r.label).collect::<Vec<_>>(),
+            ["text", "text"],
+            "dense panel must demote into one text region per paragraph"
+        );
+        assert!(regions[0].b < regions[1].t, "paragraphs split at the gap");
+        // Sparse narrow labels (a chart): picture survives.
+        let labels = vec![
+            cell("0", 5.0, 90.0, 8.0, 95.0),
+            cell("50", 5.0, 50.0, 10.0, 55.0),
+            cell("100", 5.0, 10.0, 12.0, 15.0),
+            cell("t, s", 45.0, 96.0, 55.0, 100.0),
+        ];
+        let mut regions = vec![panel];
+        super::recover_text_panels(&mut regions, &labels);
+        assert_eq!(
+            regions.iter().map(|r| r.label).collect::<Vec<_>>(),
+            ["picture"]
+        );
+    }
+
     /// A generator that paints a line's bold runs *after* its regular text
     /// hands the sanitizer the cells out of visual order ("C." | "Zur Wahrung
     /// …" | "6." — the bold label number drawn last). docling's index order

crates/docling-pdf/src/lib.rs

@@ -410,17 +410,6 @@ impl Worker {
     /// into its nodes and links. Pure given the page (mutates only the worker's
     /// lazily-loaded OCR model), so it is safe to run concurrently across pages.
     fn process(&mut self, n: usize, page: &mut PdfPage) -> Result<PageOut, PdfError> {
-        // Force-OCR is exactly "pretend the text layer is not there": clear
-        // every cell kind the extractors produced before anything reads them,
-        // and the ordinary no-text-layer machinery below — full-page OCR,
-        // OCR-fed TableFormer matching — takes over unchanged. (`no_ocr` wins
-        // when both are set, mirroring docling, where `force_full_page_ocr`
-        // is a sub-option of `do_ocr`.)
-        if self.force_full_page_ocr && !self.no_ocr {
-            page.cells.clear();
-            page.code_cells.clear();
-            page.word_cells.clear();
-        }
         if self.no_ocr {
             // Fastest path: no layout/OCR/TableFormer inference at all. The PDF's
             // embedded text cells (if any) become flat, line-grouped paragraphs in
@@ -493,6 +482,19 @@ impl Worker {
         page: &mut PdfPage,
         regions: Vec<layout::Region>,
     ) -> Result<PageOut, PdfError> {
+        // Force-OCR is exactly "pretend the text layer is not there": clear
+        // every cell kind the extractors produced before anything reads them,
+        // and the ordinary no-text-layer machinery below — full-page OCR,
+        // OCR-fed TableFormer matching — takes over unchanged. (`no_ocr` wins
+        // when both are set, mirroring docling, where `force_full_page_ocr`
+        // is a sub-option of `do_ocr`; the no-ocr path never reaches here.)
+        // Done here rather than in `process` so the batched layout path
+        // (`process_batch` → `finish_page`) honors the flag too.
+        if self.force_full_page_ocr {
+            page.cells.clear();
+            page.code_cells.clear();
+            page.word_cells.clear();
+        }
         // docling's LayoutPostprocessor drops each detection below its label's
         // confidence threshold (stricter than the 0.3 base the predictor keeps),
         // before any overlap resolution. This removes the low-confidence tables /
@@ -510,7 +512,8 @@ impl Worker {
         // remove it so it isn't emitted twice (docling parity).
         assemble::drop_contained_regulars(&mut regions);
         // No text layer → recognise text from the page image via OCR.
-        if page.cells.is_empty() {
+        let ocred = page.cells.is_empty();
+        if ocred {
             if self.ocr.is_none() {
                 self.ocr = Some(ocr::OcrModel::load(self.ocr_lang).map_err(PdfError::Ocr)?);
             }
@@ -523,6 +526,101 @@ impl Worker {
             .map_err(|e| PdfError::Ocr(format!("page {}: {e}", n + 1)))?;
             page.cells = cells;
         }
+        // Region-scoped OCR skips `picture` interiors, and a digital page's
+        // text layer cannot see into an embedded raster either — so a figure
+        // that is really a text box (terms-and-conditions exported as an
+        // image) lost its words on every page kind. Python docling OCRs the
+        // bitmap-covered areas of *every* page — even digital ones — once they
+        // exceed `bitmap_area_threshold` (5 % of the page); the browser paths
+        // already do. Recognize the big text-less crops here too; the panel
+        // demotion / orphan recovery below place the lines.
+        let mut pic_cells: Vec<pdfium_backend::TextCell> = Vec::new();
+        {
+            let page_area = (page.width * page.height).max(1.0);
+            let has_text = |r: &layout::Region| {
+                page.cells.iter().any(|c| {
+                    let ca = ((c.r - c.l) * (c.b - c.t)).max(1.0);
+                    let ix = (r.r.min(c.r) - r.l.max(c.l)).max(0.0);
+                    let iy = (r.b.min(c.b) - r.t.max(c.t)).max(0.0);
+                    !c.text.trim().is_empty() && ix * iy / ca > 0.5
+                })
+            };
+            // A captioned picture can never demote to a text panel (see
+            // recover_text_panels), and on digital pages its speculative OCR
+            // would be discarded anyway — don't pay for it.
+            let captioned = |r: &layout::Region| {
+                regions.iter().any(|c| {
+                    c.label == "caption"
+                        && c.r.min(r.r) - c.l.max(r.l) > 0.0
+                        && ((c.t >= r.b && c.t - r.b <= 25.0) || (r.t >= c.b && r.t - c.b <= 25.0))
+                })
+            };
+            let bare: Vec<layout::Region> = regions
+                .iter()
+                .filter(|r| {
+                    r.label == "picture"
+                        && (r.r - r.l) * (r.b - r.t) / page_area >= 0.05
+                        && !has_text(r)
+                        && (ocred || !captioned(r))
+                })
+                .map(|r| layout::Region {
+                    label: "text",
+                    ..r.clone()
+                })
+                .collect();
+            if !bare.is_empty() {
+                if self.ocr.is_none() {
+                    self.ocr = Some(ocr::OcrModel::load(self.ocr_lang).map_err(PdfError::Ocr)?);
+                }
+                pic_cells = timing::timed("ocr.pictures", || {
+                    self.ocr
+                        .as_mut()
+                        .unwrap()
+                        .ocr_page(&page.image, &bare, page.scale)
+                })
+                .map_err(|e| PdfError::Ocr(format!("page {}: {e}", n + 1)))?;
+                page.cells.extend(pic_cells.iter().cloned());
+            }
+        }
+        let cells_before_pic_ocr = page.cells.len() - pic_cells.len();
+        // A "picture" that is really a colored text panel — dense, wide,
+        // multi-line — reads out as paragraphs instead of shipping as pixels;
+        // sparse in-picture text (a chart's labels) keeps the crop and is
+        // emitted beside it via orphan recovery.
+        assemble::recover_text_panels(&mut regions, &page.cells);
+        // On an OCR'd page, in-picture text that did NOT demote its picture is
+        // still emitted beside the kept crop (matching the browser scanned
+        // path). On a digital page it is not: docling's groundtruth keeps
+        // photos silent even when our OCR reads noise off them
+        // (picture_classification stays byte-exact), so the recognized cells
+        // there only ever serve the panel-demotion decision above.
+        if ocred && !pic_cells.is_empty() {
+            let mut probe: Vec<layout::Region> = regions
+                .iter()
+                .filter(|r| r.label != "picture")
+                .cloned()
+                .collect();
+            let keep_from = probe.len();
+            assemble::add_orphan_regions(&mut probe, &pic_cells);
+            regions.extend(probe.drain(keep_from..));
+        } else if !ocred && !pic_cells.is_empty() {
+            // Digital page, picture kept: its speculative OCR cells must not
+            // linger in the text-cell set (they were appended at the tail).
+            let kept: Vec<layout::Region> = regions
+                .iter()
+                .filter(|r| r.label == "picture")
+                .cloned()
+                .collect();
+            let tail = page.cells.split_off(cells_before_pic_ocr);
+            page.cells.extend(tail.into_iter().filter(|c| {
+                !kept.iter().any(|r| {
+                    let ca = ((c.r - c.l) * (c.b - c.t)).max(1.0);
+                    let ix = (r.r.min(c.r) - r.l.max(c.l)).max(0.0);
+                    let iy = (r.b.min(c.b) - r.t.max(c.t)).max(0.0);
+                    ix * iy / ca > 0.5
+                })
+            }));
+        }
         // TableFormer structure per table region (else geometric fallback). The
         // shared slot is only locked (and lazily loaded) when the page actually
         // has a table, so table-free documents never pay for TableFormer at all.