text-extraction: combining diacritics emitted adjacent to base letter (´Ecole, Sup erieure,´) and CJK parens displaced

[yfedoseev]

Summary

Combining diacritical marks (acute, grave, circumflex, cedilla) emitted by pdfTeX-typeset documents are placed adjacent to the base letter rather than over it, frequently after a space. Latin small letters become two-glyph sequences like ´E, Sup erieure,´, Universit e´ (instead of É, Supérieure, Université). Mirror behaviour is visible for CJK parenthesised author names (see below).

Reproduction

import pdf_oxide  # 0.3.54
doc = pdf_oxide.PdfDocument("2201.00200.pdf")  # arXiv:2201.00200
print(doc.extract_text(0).split('\n')[13])
# '2 ´Ecole Normale Sup erieure,´ Lyon, CRAL (UMR CNRS 5574), Universit e´ de Lyon, France'

PDF: https://arxiv.org/pdf/2201.00200.pdf

For CJK author names in https://arxiv.org/pdf/2201.00021.pdf:

expected:  Y. T. Yan (闫耀庭)1
actual:    Y. T. Yan 闫( 耀庭) 1

— the opening parenthesis is moved across the CJK character.

Expected

2 École Normale Supérieure, Lyon, CRAL (UMR CNRS 5574), Université de Lyon, France
... Y. T. Yan (闫耀庭)1 ...

Actual

2 ´Ecole Normale Sup erieure,´ Lyon, CRAL (UMR CNRS 5574), Universit e´ de Lyon, France
... Y. T. Yan 闫( 耀庭) 1 ...

Impact

• Levenshtein loss on 2201.00200 (75% → could recover ~5pt) and any paper with French/Czech/Vietnamese author affiliations or CJK names.
• Breaks string search for author names.
• The CJK paren reordering is the same root cause as the Latin combining-mark mis-placement — both arise from emitting glyph runs in glyph-positional order without coalescing combining-class characters onto the preceding base.

Hypothesis

The text extractor walks the content stream's glyph positions and emits Unicode in encounter order without:

1. Normalizing combining sequences (NFC), and
2. Detecting that a glyph at position (x,y) whose width is ~0 and whose Unicode is in combining class > 0 belongs to the next or previous base glyph.

This is similar to the BiDi handling for RTL — combining marks need their own positional reorder pass.

Suggested fix

After glyph-to-Unicode mapping but before line assembly, run NFC normalization on each glyph run, and detect zero-width combining marks placed on a separate text-positioning operator (Td/TJ) — collapse them onto the lexically adjacent base character.

Related

• Add pdf_oxide to benchmark suite py-pdf/benchmarks#19
• Adjacent to Multi-column prose reading order: tight two-column layouts interleave row-by-row #534 / XY-cut: multi-line headings adjacent to a dense column get split (follow-up to #534) #543 (reading-order pipeline)