text-extraction: intra-word space inserted in narrow-tracked / justified columns (Exp erimen ts, Departmen t, kno wledge)

[yfedoseev]

Summary

In narrow-tracked / justified-column PDFs (especially 1990s academic typesetting), pdf_oxide v0.3.54 splits visually contiguous words at near-zero intra-glyph gaps, producing fragments like Exp erimen ts, Departmen t, Univ ersit y, kno wledge, Bac kground, tec hniques, Pro v en.

This is distinct from #551 (ligature decomposition) — the fragments are at arbitrary positions within ordinary roman letters, not at fi/fl/ffi boundaries. It is also distinct from #555 (missing space at font/run boundary) — here the issue is the opposite: a space is INSERTED inside a word, not omitted between words.

The hypothesis is glyph-cluster boundary detection: pdf_oxide's word-segmentation threshold is calibrated for modern typography (~0.25 em gap = inter-word space), but tightly kerned justified columns can have intra-word advance widths that briefly exceed that threshold (e.g. between a wide letter and a narrow one in a stretched paragraph).

Reproduction

import pdf_oxide  # 0.3.54
doc = pdf_oxide.PdfDocument("09f90a8fad0997f7cf454cbcbe79cab3bc0f_page_1_pg1.pdf")
words = doc.extract_words(0)
print([w.text for w in words[:30]])
# ['Experimen', 'Sanda', 'M.', 'This', 'paper', 'describes', 'kno', 'wledge-basednatural',
#  'niques', 'in', 'to', 'a', 'Question', 'of', 'mining', 'textual', 'texts.',
#  'Surprizing', 'ligh', 't', 'w', 'eigh', 't', 'kno', 'wledge-based', 'plemen', 't', ...]

PDF: multi_column/09f90a8fad0997f7cf454cbcbe79cab3bc0f_page_1_pg1.pdf from allenai/olmOCR-bench (a 1990s open-domain QA paper).

Expected

Experiments with Open-Domain Textual Question Answering
Sanda M. Harabagiu and Marius A. Pasca and Steven J. Maiorano
Department of Computer Science and Engineering
Southern Methodist University
... This paper describes the integration of several knowledge-based natural
language processing techniques into a Question Answering system, capable
of mining textual answers from large collections of texts. Surprising quality
is achieved when several lightweight knowledge-based NLP techniques ...

Actual

Exp erimen ts with Op en-Domain T extual Question Answ ering
Sanda M. Harabagiu and Marius A. P a sca and Stev en J. Maiorano
Departmen t of Computer Science and Engineering
Southern Metho dist Univ ersit y
... This pap er describ es the in tegration of sev eral
kno wledge-based natural language pro cessing tec h-
niques in to a Question Answ ering system, capable
of mining textual answ ers from large collections of
texts. Surprizing qualit y is ac hiev ed when sev eral
ligh t w eigh t kno wledge-based NLP tec hniques com-

Impact

Scanned all 211 PDFs in allenai/olmOCR-bench multi_column subset; 5 PDFs (~2.4%) exhibit ≥3 confidently-detected intra-word splits, with the worst showing 10+ splits per page. Total assertion failures on those PDFs is 100% (every order check fails because the split words can't be matched as substrings of the ground-truth phrases).

Affected PDFs (sample):

• multi_column/09f90a8fad0997f7cf454cbcbe79cab3bc0f_page_1_pg1.pdf
• multi_column/09f90a8fad0997f7cf454cbcbe79cab3bc0f_page_7_pg1.pdf
• multi_column/0097571b436308f563a5808d45d0232255a2_page_6_pg1.pdf
• multi_column/01460d3dc3eca92c0d1a17eff846fa682f3d_page_6_pg1.pdf

Related to #555 (the inverse — missing inter-word space) and likely shares the same threshold heuristic.

Hypothesis

The horizontal-gap heuristic that promotes a glyph-cluster boundary to a word boundary likely uses an absolute (or font-size-relative) threshold without normalising for the line's justification stretch factor. In stretched justified columns, the per-glyph advance is increased uniformly, so intra-word gaps cross the threshold. Calibration on the median intra-word gap of the line (or font's space advance) before promoting boundaries would address this.

Bench corpus

https://huggingface.co/datasets/allenai/olmOCR-bench (license: ODC-BY)

Cross-ref: py-pdf/benchmarks#19