feat(text): script detection, normalisation, bidi shaping and glyph coverage - #3
Merged
Merged
Conversation
…ph coverage Arabic ground truth has two representations that must never be confused: the logical string the model predicts and the visual string the rasteriser draws. ocrsmith.text makes that split explicit. - script.py: script classification and first-strong-character base direction. - normalization.py: NormalizationPolicy with opt-in, idempotent transforms (diacritics, tatweel, alef/ya/ta-marbuta unification, numeral systems, whitespace). Every transform changes the label, so none is applied silently. - shaping.py: two interchangeable backends behind one protocol. Pillow builds with Raqm shape and reorder themselves (TransparentShaper); builds without it get presentation forms and visual reordering from arabic-reshaper + python-bidi (ReshaperBidiShaper). Both keep the logical label identical, so the dataset does not depend on how Pillow was compiled. - coverage.py: exact cmap coverage via fontTools, cached per font file. A font that cannot draw a character is now rejectable instead of silently rendering tofu that contradicts the label. Adds fonttools as a dependency and drops two empty placeholder modules. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
2 of 9 in the v1.0.0 series. Base:
chore/dev-tooling.Why
Arabic ground truth has two representations that must never be confused: the logical string a model predicts, and the visual string the rasteriser draws. Mixing them up is the single most common way to produce an unusable Arabic OCR dataset, and it is invisible until training plateaus.
What
New
ocrsmith.textpackage.script.py— script classification and first-strong-character base direction.normalization.py—NormalizationPolicy: opt-in, idempotent transforms for diacritics, tatweel, alef/ya/ta-marbuta unification, numeral systems (Western / Arabic-Indic / Eastern), whitespace. Every one of these rewrites the label, so none is applied silently.shaping.py— two interchangeable backends behind one protocol:TransparentShaperwhen Pillow was built with Raqm (HarfBuzz shapes and reorders);ReshaperBidiShaperotherwise (arabic-reshaper + python-bidi produce presentation forms and visual order).Both keep the logical string as the label, so the dataset does not depend on how Pillow was compiled.
coverage.py— exact glyph coverage read from the font'scmapvia fontTools, cached per file. A font that cannot draw a character becomes rejectable instead of silently rendering tofu that contradicts the label.Adds
fonttools; drops two empty placeholder modules.Verified
106 tests green. Coverage tests run against the bundled Arabic fonts.