Skip to content

fix(pdf): bridge unassigned decimal digits in native span fill - #5518

Open
kabishou11 wants to merge 1 commit into
opendatalab:masterfrom
kabishou11:fix/5517-bridge-unassigned-digits
Open

kabishou11 wants to merge 1 commit into
opendatalab:masterfrom
kabishou11:fix/5517-bridge-unassigned-digits

Conversation

@kabishou11

Copy link
Copy Markdown

Problem

Native PDF span fill can drop decimal digits (ASCII or fullwidth Nd) when mixed CJK/digit fonts place those glyphs slightly off the layout span mid-line. Body text then loses the numeric run while nearby punctuation is still recovered, producing strings such as 至 . 个 / 大约 个百分点. OCR mode is unaffected because it does not use this native assignment path.

This matches the CNKI-style field report in #5330.

Root cause

After primary tight/loose calculate_char_in_span assignment, _bridge_unassigned_punctuation recovered only Unicode category P* glyphs, and only when both visible neighbors already belonged to the same span and the glyph center was still inside that span.

Decimal digits (Nd) that fail the center-axis height-ratio check were never bridged, so chars_to_content never saw them. The same gap's punctuation could still be recovered.

Fix

Extend the existing neighbor-agree + center-inside-span bridge to single-codepoint decimal digits (unicodedata.category(ch) == "Nd") as well as P*. Safety gates are unchanged:

  • both previous/next visible owners assigned and equal
  • center inside the owner bbox via tight-then-loose
  • no cross-line bridging (CONTROL_LINE_BREAK_CHARS still clears neighbor owners)

Letters (L*) are not bridged. calculate_char_in_span thresholds are not loosened. The helper is renamed to _bridge_unassigned_inline_glyphs / _is_bridgeable_unassigned_char.

Test evidence

Focused tests in tests/unittest/test_pdf_native_script_detection.py:

  • fullwidth 至0.73个 with CJK neighbors that pass geometry and intervening 0.73 that fail calculate_char_in_span now restores 至0.73个 (failed as 至 . 个 before the fix)
  • same geometry with ASCII 至0.73个 restores 至0.73个 (failed as 至 . 个 before the fix)
  • unassigned digit between two different spans is not absorbed
  • existing punctuation bridge tests remain green
.venv/bin/python -m pytest -q -o addopts='' \
  tests/unittest/test_pdf_native_script_detection.py \
  -k 'bridge or fullwidth or digit or span_fill_bridges or span_fill_does_not_bridge'

9 passed (2 failed before the production change).

Related to #5517
Related to #5330

Recover unassigned Nd digits during native span fill when both visible
neighbors already belong to the same span and the glyph center remains
inside that span. Punctuation was already bridged; mixed-font decimals
were dropped.

Related to opendatalab#5517
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@github-actions

github-actions Bot commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@kabishou11

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

github-actions Bot added a commit that referenced this pull request Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant