fix(pdf): bridge unassigned decimal digits in native span fill - #5518
Open
kabishou11 wants to merge 1 commit into
Open
kabishou11 wants to merge 1 commit into
kabishou11 wants to merge 1 commit into
Conversation
Recover unassigned Nd digits during native span fill when both visible neighbors already belong to the same span and the glyph center remains inside that span. Punctuation was already bridged; mixed-font decimals were dropped. Related to opendatalab#5517
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Contributor
|
All contributors have signed the CLA ✍️ ✅ |
Author
|
I have read the CLA Document and I hereby sign the CLA |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Native PDF span fill can drop decimal digits (ASCII or fullwidth
Nd) when mixed CJK/digit fonts place those glyphs slightly off the layout span mid-line. Body text then loses the numeric run while nearby punctuation is still recovered, producing strings such as至 . 个/大约 个百分点. OCR mode is unaffected because it does not use this native assignment path.This matches the CNKI-style field report in #5330.
Root cause
After primary tight/loose
calculate_char_in_spanassignment,_bridge_unassigned_punctuationrecovered only Unicode categoryP*glyphs, and only when both visible neighbors already belonged to the same span and the glyph center was still inside that span.Decimal digits (
Nd) that fail the center-axis height-ratio check were never bridged, sochars_to_contentnever saw them. The same gap's punctuation could still be recovered.Fix
Extend the existing neighbor-agree + center-inside-span bridge to single-codepoint decimal digits (
unicodedata.category(ch) == "Nd") as well asP*. Safety gates are unchanged:CONTROL_LINE_BREAK_CHARSstill clears neighbor owners)Letters (
L*) are not bridged.calculate_char_in_spanthresholds are not loosened. The helper is renamed to_bridge_unassigned_inline_glyphs/_is_bridgeable_unassigned_char.Test evidence
Focused tests in
tests/unittest/test_pdf_native_script_detection.py:至0.73个with CJK neighbors that pass geometry and intervening0.73that failcalculate_char_in_spannow restores至0.73个(failed as至 . 个before the fix)至0.73个restores至0.73个(failed as至 . 个before the fix)9 passed (2 failed before the production change).
Related to #5517
Related to #5330