text-extraction: U+FFFD chars from glyph-name-only fonts (MSAM10, dingbats) returned by extract_chars but silently dropped by extract_text/words/spans

[yfedoseev]

Summary

When a PDF uses a font that has no Unicode mapping (e.g. TeX's MSAM10 AMS math-symbols font, dingbats, or any font with no ToUnicode CMap and only glyph names), pdf_oxide decodes its glyphs to U+FFFD (REPLACEMENT CHARACTER). That's fine in isolation — but the three text-extraction APIs disagree about what to do with those chars:

API	Returns FFFD chars?
extract_chars(page)	yes (one TextChar per glyph, .char == '�')
extract_words(page)	silently drops them (returns 0 words)
extract_text(page)	silently drops them (returns "")
extract_spans(page)	silently drops them (returns 0 spans)

The page has visible glyphs and extract_chars correctly surfaces them, but the higher-level text APIs filter U+FFFD and leave the caller with no signal that this page contains un-mapped glyphs. Symptom is indistinguishable from extract_text on an empty / image-only page (#563), but the underlying cause is different.

Reproduction

Fixture: bug1068432.pdf from mozilla/pdf.js test corpus (MIT).
Download: https://github.com/mozilla/pdf.js/blob/master/test/pdfs/bug1068432.pdf?raw=true

import pdf_oxide  # 0.3.54
d = pdf_oxide.PdfDocument("bug1068432.pdf")

print(len(d.extract_chars(0)))   # -> 3
for c in d.extract_chars(0):
    print(repr(c.char), c.font_name)
# '�' AAAAAN+MSAM10
# '�' AAAAAN+MSAM10
# '�' AAAAAN+MSAM10

print(repr(d.extract_text(0)))   # -> ''
print(len(d.extract_words(0)))   # -> 0
print(len(d.extract_spans(0)))   # -> 0

Rendered, the page shows three visible math arrows: ↑↑ ↓↓ (drawn from MSAM10).

Expected

Either:

1. The three text APIs consistently emit U+FFFD for glyphs without a Unicode mapping (so callers can detect them and decide whether to drop, OCR, or surface them), or
2. They consistently drop U+FFFD and expose some per-page signal (e.g. page_has_unmapped_glyphs() or a warning in flatten_warnings()) so the caller can distinguish "no text" from "text that pdf_oxide chose to hide".

Right now the divergence is invisible: anyone going through extract_text gets "" and a page_count() > 0 with no indication that they're missing content.

Actual

extract_chars returns 3 TextChar objects, the other three APIs return empty / zero, no warning is logged in this case.

Hypothesis

Whatever string-building stage feeds extract_text / extract_words / extract_spans filters out '�' chars (probably as a "looks like garbage" heuristic) before joining. extract_chars lives one layer below that filter and survives.

Impact

Math-heavy PDFs (TeX/LaTeX), symbol/dingbat fonts, decorative drop caps, and any PDF whose font lacks ToUnicode + only has glyph names will silently produce empty text from the high-level APIs even when the page has visible content. This is a different root cause from #563 (image-only pages) — those have zero TextChars, this one has 3 — and the symptoms are easy to confuse.

Related

• text-extraction: image-only / rasterized PDFs return empty string with no signal that the page has no text layer #563 (image-only PDFs return empty without signal) — adjacent symptom, different root cause
• text-extraction: three plain-text APIs (extract_text / to_plain_text / markdown-strip) produce inconsistent output; best mode varies per PDF #554 (three plain-text APIs inconsistent) — adjacent inconsistency, between extract_text / to_plain_text / to_markdown; this issue is between extract_chars and the rest
• Add pdf_oxide to benchmark suite py-pdf/benchmarks#19