[Bug]: extraction-correctness bugs on the text/table/image path (each verified against pdftotext / PyMuPDF / pdfminer)

[tobocop2]

A focused pass over the code reached by from_bytes → to_markdown / to_plain_text, table extraction, image extraction, forms, and link annotations. Companion to #994, which covered the non-extraction subsystems.

Note

One issue as an epic, sorry for the format — these are all independent extraction bugs, split off whatever's useful. Every finding is wrong or lost text / table / image output on a valid PDF.

Important

Runnable reproducers for all 12 findings are in this gist: https://gist.github.com/tobocop2/3c1388fbdb338c62efca164d4db872e2 — the integration tests plus the PDF generators that build each crafted fixture, keyed to the table below.

How each was verified

Each finding was reproduced differentially: the same PDF is extracted with pdf_oxide and with a reference extractor (pdftotext/poppler, PyMuPDF, and/or pdfminer/pypdf), and reported only where pdf_oxide produces the wrong output and a reference produces the right one — or, for crafted inputs, where pdf_oxide disagrees with the file's known-correct content. Where every extractor agreed, or the file was undecodable, it was dropped. Run against v0.3.77.

Comparison table

Read each row as: given the Trigger, pdf_oxide outputs the pdf_oxide column, but the correct output (per the reference extractors and the PDF spec) is the Correct column.

#	Sev	Location	Trigger	pdf_oxide output	Correct output
1	High	src/fonts/cid_mappings/adobe_japan1.rs:2333	Japanese 青 shown via an Adobe-Japan1 CIDFont (CID 2664), no ToUnicode	⻘ (U+2ED8, a radical presentation form)	青 (U+9752, the real ideograph)
2	Medium	src/fonts/font_dict.rs:5154	any radical-form char in U+2E80–2EFF (e.g. ⻘) passing the "normalize radicals" guard	⻘ unchanged — the guard silently no-ops	青 (U+9752)
3	Low	src/extractors/text.rs:8427	Type0/Identity-H run 50 Tw <0020 0021> Tj (the 2-byte code 0x0020 is a real glyph, not a space)	span right edge x1 = 148 (word-spacing wrongly applied)	x1 = 98 — matches the TJ form, PyMuPDF, and ISO 32000-1 §9.3.3
4	High	src/pipeline/converters/mod.rs:359	20pt heading "Revenue" then 12pt paragraph "2024 was a record year." (two baselines)	## Revenue 2024 was a record year. (heading swallows the paragraph)	Revenue heading + separate 2024 was a record year. paragraph
5	Medium	src/pipeline/converters/markdown.rs:350	three distinct 20pt headings "Sales", "Marketing", "Engineering"	## Sales Marketing Engineering (one heading)	three separate headings
6	Medium	src/pipeline/converters/markdown.rs:610	bold text "A gitHub repo" (single upper word + CamelCase word + lower word)	**AgitHubrepo** (both spaces deleted)	A gitHub repo
7	High	src/structure/table_extractor.rs:829	a table cell containing two runs "Hello" then "World" under one MCID	cell = Hello; "World" is dropped out of the table entirely	cell keeps Hello World
8	High	src/extractors/images.rs:954	a 1-bit DeviceGray 8×8 image	8 packed bytes labelled 8-bit Grayscale (buffer too short to be a valid 8×8 plane)	64 unpacked samples [0,255,0,255,…]
9	Medium	src/extractors/images.rs:942	an 8-bit gray image with /Decode [1 0] (tonal invert)	raw samples [0,1,2,3,…] — /Decode ignored	inverted [255,254,253,…]
10	High	src/encryption/algorithms.rs:242	an AES-256 R5 encrypted PDF, empty user password	authenticate() returns true but extract_text is "" (streams decrypt to noise)	the real text (the R6 path handles the identical file correctly)
11	Medium	src/annotations.rs:653	an internal link to page 2, whose page object is number 6	destination page index 6 (the raw PDF object number)	page index 1 (0-based position in the page tree)
12	Medium	src/extractors/forms.rs:548	a radio/checkbox with /V /No selected (a legitimate on-state)	Boolean(false) — indistinguishable from unset/off	the export value No

_{References: pdftotext (poppler 25.06), PyMuPDF (fitz), pdfminer/pypdf. "Correct output" is the value all references agree on, matching the ISO 32000-1 spec where cited.}

Details & reproducers

One entry per row above — the root cause and the exact command that reproduces the comparison.

1. High — src/fonts/cid_mappings/adobe_japan1.rs:2333

The Adobe-Japan1 CID→Unicode table maps common ideograph CIDs to CJK Radicals Supplement codepoints (U+2E80–2EFF) instead of the unified ideographs (CID 2664→U+2ED8 instead of U+9752 青; CID 2666→U+2EEB instead of U+6589 斉). Predefined CJK CIDFonts without a ToUnicode map therefore extract the wrong character. Cross-checked against pdfminer's own Adobe-Japan1 map: cid2unichr[2664]='青'.

Entry point: to_plain_text / to_markdown
Reproduce: cargo test --test zzz_xaudit_fonts-glyph-unicode japan1_cidfont_radical_forms -- --nocapture

2. Medium — src/fonts/font_dict.rs:5154

normalize_cjk_radical_forms is documented (lines 5140–5148) to carry radical presentation forms in both the Supplement (U+2E80–2EFF) and Kangxi (U+2F00–2FDF) blocks to ideographs via NFKC — but NFKC has decompositions for only 2 of the 128 Supplement-block codepoints, so the guard does nothing for that entire block and merely masks (rather than fixes) the Kangxi-block table entries. This is the layer that should have caught finding 1 and doesn't.

Entry point: to_plain_text / to_markdown (runs on every char)
Reproduce: cargo test --test zzz_xaudit_fonts-glyph-unicode japan1_cidfont_radical_forms -- --nocapture

3. Low — src/extractors/text.rs:8427

The plain-Tj path applies word spacing (Tw) to a 2-byte code whose CID is 0x20 by testing char_code == 32 with no byte-width guard; the sibling TJ helper guards correctly with nbytes == 1 && cid == 32, so the two paths disagree on identical content. Note: the text ("AB") is unaffected — only the reported span width — because any later reposition resets the text matrix.

Entry point: extract_spans → TextSpan.bbox
Reproduce: cargo test --test zzz_xaudit_content-interp-spans -- --nocapture

4. High — src/pipeline/converters/mod.rs:359

merge_key_value_pairs treats a font-size heading as a key/label and glues the following short paragraph onto the same line whenever that paragraph starts with a digit, $, (, or -, producing a malformed heading and destroying the paragraph break.

Entry point: to_markdown / to_plain_text
Reproduce: cargo test --test zzz_xaudit_markdown-readingorder heading_absorbs_following_numeric_paragraph -- --nocapture

5. Medium — src/pipeline/converters/markdown.rs:350

The three_plus_short policy of merge_consecutive_same_level_headings fuses any run of 3+ consecutive same-level headings that are each ≤2 words into a single heading line, collapsing genuinely distinct section headers.

Entry point: to_markdown
Reproduce: cargo test --test zzz_xaudit_markdown-readingorder three_short_headings_wrongly_merged -- --nocapture

6. Medium — src/pipeline/converters/markdown.rs:610

coalesce_camelcase_bold_fragments deletes the spaces inside any bold run shaped **A wordCap word**, silently concatenating legitimate bold prose into one token.

Entry point: to_markdown
Reproduce: cargo test --test zzz_xaudit_markdown-readingorder camelcase_bold_eats_spaces -- --nocapture

7. High — src/structure/table_extractor.rs:829

extract_cell's inner-loop break keeps only the first text block matching each cell's MCID; every later span sharing that TD's MCID is silently dropped from the TableCell, contradicting the line-690 comment ("collect all text blocks that match these MCIDs"). Multi-run cells lose everything after their first run.

Entry point: to_markdown with table extraction
Reproduce: cargo test --test zzz_xaudit_tables crafted_tagged_pdf_to_markdown_cell_loses_world -- --nocapture

8. High — src/extractors/images.rs:954

Sub-8-bit non-CCITT images (1/2/4-bpc DeviceGray, and any sub-byte non-DeviceGray) fall through the else { decoded_data } arm and are returned as the packed sub-byte stream while labelled PixelFormat::Grayscale/RGB, whose bytes_per_pixel implies unpacked 8-bit samples. The length no longer matches w*h*bpp, so a consumer that builds an image from the buffer rejects it and the image is lost. Both PyMuPDF and poppler return the full 64-sample unpacked plane.

Entry point: extract_images
Reproduce: cargo test --test zzz_xaudit_image-extract-ocr --features rendering one_bit_devicegray_emits_packed_buffer -- --nocapture

9. Medium — src/extractors/images.rs:942

A non-default /Decode array is honored only for 1-bit DeviceGray and CCITT. For BitsPerComponent ≥ 8 (and non-DeviceGray) the else arm returns the samples verbatim, so an inverted-/Decode image is extracted with tonally-wrong pixels. ISO 32000-1 §8.9.5.2 requires samples to be mapped through /Decode before use.

Entry point: extract_images
Reproduce: cargo test --test zzz_xaudit_image-extract-ocr --features rendering decode_array_ignored_for_8bit_gray -- --nocapture

10. High — src/encryption/algorithms.rs:242

For AES-256 R5 the file key is taken as SHA256(password‖key_salt) and returned directly instead of AES-256-CBC-decrypting the /UE entry with that intermediate key. R6 (lines 233–241) unwraps /UE correctly; R5 (lines 243–248) does not — so authentication succeeds but every stream decrypts to garbage and extraction silently yields empty output.

Entry point: from_bytes → extract_text / to_markdown
Reproduce: cargo test --test zzz_xaudit_forms-xfa-annots-decrypt finding1_r5_aes256_decrypt -- --nocapture

11. Medium — src/annotations.rs:653

parse_destination for an explicit [pageRef /XYZ …] destination uses the raw object number as the page index instead of resolving the reference to its position in the page tree, so link targets point at the wrong page (out of range when the object number exceeds the page count). The code comment says "For now, just use object ID as approximation."

Entry point: get_annotations → link destination
Reproduce: cargo test --test zzz_xaudit_forms-xfa-annots-decrypt finding2_link_goto_page_index -- --nocapture

12. Medium — src/extractors/forms.rs:548

parse_field_value maps a Button field whose /V name is "No" or "Off" to Boolean(false), collapsing an actively-selected export value into a boolean; a selected "No" becomes indistinguishable from an unchecked field even though "No" is a real on-state.

Entry point: form field extraction → FieldValue
Reproduce: cargo test --test zzz_xaudit_forms-xfa-annots-decrypt finding3_button_no_export_value -- --nocapture

---

_{Reproducers (integration tests + the PDF generators for each crafted fixture): https://gist.github.com/tobocop2/3c1388fbdb338c62efca164d4db872e2}