text-extraction: three plain-text APIs (extract_text / to_plain_text / markdown-strip) produce inconsistent output; best mode varies per PDF

[yfedoseev]

Summary

The three "plain text" public APIs on PdfDocument (extract_text(page), to_plain_text(page) / to_plain_text_all(), and the markdown→strip path via to_markdown_all()) produce substantially different output on the same input, with no documentation of when to use which. None is uniformly best — best mode depends on the PDF.

Measured behaviour (py-pdf/benchmarks corpus, 14 PDFs, Levenshtein vs ground truth)

PDF	extract_text	to_plain_text_all	markdown-stripped	best mode
2201.00214	95.5	94.3	89.2	extract_text
GeoTopo-book	91.3	89.4	73.6	extract_text
2201.00151	52.8	62.7	62.9	md/plain ≈ tie
1707.09725	92.3	94.8	83.9	to_plain_text_all
2201.00021	66.8	63.1	90.0	markdown
2201.00037	92.6	93.6	86.8	to_plain_text_all
2201.00069	84.5	64.5	92.0	markdown
2201.00178	87.7	90.6	85.6	to_plain_text_all
2201.00201	85.4	59.5	95.9	markdown
1602.06541	76.7	59.3	92.4	markdown
2201.00200	75.3	69.5	94.8	markdown
2201.00022	63.1	58.6	73.1	markdown
2201.00029	85.4	96.1	89.8	to_plain_text_all
1601.03642	74.0	67.2	93.0	markdown
mean	80.2	75.9	85.9	best-of-each: 89.7

Why this matters

• A consumer who picks extract_text (the most discoverable name) loses ~10 pt of accuracy on average vs to_markdown_all-and-strip.
• A consumer who picks to_plain_text_all loses ~20 pt on column-heavy PDFs.
• A simple plain-text caller has no way to know which mode their PDF wants; the best-of-each strategy is +9.5 pt over extract_text.

Hypothesis

Three different pipelines exist internally (legacy glyph-walk for extract_text, an intermediate path for to_plain_text, the XY-cut markdown converter for to_markdown_all) and the inconsistency reflects partial rollout of the #534 fix.

Suggested fix

Either:

1. Unify all plain-text outputs onto the XY-cut markdown pipeline followed by markdown stripping, OR
2. Document explicitly which API is "high fidelity / structure-aware" and have extract_text delegate to it by default, with a legacy=True flag for backward compatibility.

Optional additional surface: expose a to_plain_text_high_fidelity() that does markdown→strip internally, so downstream code doesn't have to know about the markdown layer.

Reproduction

import pdf_oxide, re
from Levenshtein import ratio
doc = pdf_oxide.PdfDocument("2201.00201.pdf")
gt = open("2201.00201_gt.txt").read()
print("extract_text:", ratio(gt, '\n'.join(doc.extract_text(i) for i in range(doc.page_count()))))
print("to_plain_text_all:", ratio(gt, doc.to_plain_text_all()))
md = re.sub(r'[#*_`>]+', '', doc.to_markdown_all())
print("md-stripped:", ratio(gt, md))

Related

• Add pdf_oxide to benchmark suite py-pdf/benchmarks#19
• text-extraction: extract_text / to_plain_text bypass XY-cut and interleave columns row-by-row (9/14 academic PDFs, ~10–23 pt accuracy loss) #549 (column interleave in extract_text)
• Multi-column prose reading order: tight two-column layouts interleave row-by-row #534, XY-cut: multi-line headings adjacent to a dense column get split (follow-up to #534) #543 (XY-cut)