Skip to content

Commit 07af462

Browse files
artizclaude
andcommitted
docling-pdf: pin the browser word-segmentation threshold with a test
Investigating a cell-merge report from the browser pipeline ("Telefon" and "0676/2000" landing in one TableFormer cell) meant knowing how coarse the OCR-derived word boxes actually are, since those are the one input the browser path does not share with the native one (no pdfium text layer — words come from segment_words' ink projection). Measured: the split happens at a gap of 0.57-0.60 x line height, consistently across line sizes. That is the floor on how tight a table's columns may be before two entries become a single box — and a single box can only ever match one table cell. Pin it, so the constant is not silently retuned. Refs #157 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
1 parent 3defa1c commit 07af462

1 file changed

Lines changed: 45 additions & 0 deletions

File tree

crates/docling-pdf/src/ocr_prep.rs

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -435,3 +435,48 @@ mod tests {
435435
assert_eq!(decode_row(&chars, &probs, 4), "ab");
436436
}
437437
}
438+
439+
#[cfg(test)]
440+
mod word_segmentation {
441+
use image::{Rgb, RgbImage};
442+
443+
/// A white line carrying two ink blocks separated by `gap` pixels.
444+
fn line_with_gap(h: u32, gap: u32) -> RgbImage {
445+
let w = 30 + gap + 30 + 10;
446+
let mut img = RgbImage::from_pixel(w, h, Rgb([255, 255, 255]));
447+
for (x0, x1) in [(5u32, 35u32), (35 + gap, 65 + gap)] {
448+
for x in x0..x1.min(w) {
449+
for y in h / 4..(3 * h / 4) {
450+
img.put_pixel(x, y, Rgb([0, 0, 0]));
451+
}
452+
}
453+
}
454+
img
455+
}
456+
457+
/// [`segment_words`] splits on a gap of `0.6 x line height`, which is the
458+
/// property the rest of the browser table path inherits: an inter-word
459+
/// space never splits, but a column gap wider than that does. Anything
460+
/// narrower stays a single box — and a single box can only ever land in one
461+
/// TableFormer cell, so this threshold is the floor on how tight a table's
462+
/// columns may be before its cells merge.
463+
#[test]
464+
fn words_split_only_on_gaps_above_six_tenths_of_the_line_height() {
465+
for h in [16u32, 24, 32, 40] {
466+
let split_at = (1..=40u32)
467+
.find(|&gap| super::segment_words(&line_with_gap(h, gap)).len() >= 2)
468+
.expect("some gap splits");
469+
let ratio = split_at as f32 / h as f32;
470+
assert!(
471+
(0.5..=0.65).contains(&ratio),
472+
"h={h}: split at {split_at}px ({ratio:.2} x height)"
473+
);
474+
// Just below the threshold the two blocks are one word.
475+
assert_eq!(
476+
super::segment_words(&line_with_gap(h, split_at - 1)).len(),
477+
1,
478+
"h={h}: a narrower gap must not split"
479+
);
480+
}
481+
}
482+
}

0 commit comments

Comments
 (0)