feat(text): vary diacritics per document without inventing them - #14
Open
hBouanane wants to merge 1 commit into
Open
feat(text): vary diacritics per document without inventing them#14hBouanane wants to merge 1 commit into
hBouanane wants to merge 1 commit into
Conversation
Arabic OCR handles vocalisation badly and it is the first limitation AtlasOCR reports about itself. The cause is distributional: real Arabic is partially diacritised and the proportion varies by genre, so a corpus that is uniformly marked or uniformly bare teaches a model to expect that uniformity. DiacriticsPolicy samples per document across keep/strip/partial/mixed and records the kept fraction in provenance, which is what makes a diacritics ablation possible later. DatasetStats reports the corpus split across bare, partial and fully marked pages. Marks are only ever removed, never added. Diacritising bare text requires a diacritiser model and would make the label assert vowels nobody wrote - a fabricated ground truth that looks entirely plausible. The tests pin that guarantee: no mode may add a mark, and the consonant skeleton is never touched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the limitation AtlasOCR states first about itself: "limited accuracy with diacritized text, as training focused on undiacritized material."
The design decision worth reviewing
Marks are only ever removed, never invented.
The obvious approach — run a diacritiser over bare text to create vocalised training data — would make the label assert vowels nobody wrote. That is fabricated ground truth, and it looks entirely plausible until a model trained on it starts hallucinating vowels.
Instead: start from a diacritised corpus and remove a sampled fraction. Removal is always truthful, because the remaining text is a form the source actually supports.
Why varying it matters
Real Arabic is partially diacritised, and the proportion varies by genre — religious and pedagogical texts fully marked, news lightly marked, most prose bare. A corpus that is uniformly one or the other teaches a model to expect that uniformity.
Also
diacritics_keptper sample, so a diacritics ablation is possible without regenerating.DatasetStatsreports the corpus split acrossnone/partial/full, because the interesting question is compositional, not an average.diacritic_ratio()for measuring what a corpus actually contains rather than what it was assumed to.Verified
500 tests (17 new). The strongest test asserts the guarantee directly: across every mode and seed, no mark is ever added to bare text, and a partial strip never alters the consonant skeleton.