mai: port Maithili to C#, and pin the divergences the golden cannot see - #1033
Merged
Conversation
Maithili runs on the shared `makeNativeHindi` abugida engine, so the port is 39 lines of
data-wiring plus one thing no sibling has: a `normalize` override that folds ⟨॑⟩ U+0951 onto
the avagraha ⟨ऽ⟩. 200/200 first run; the full gate goes 82 → 83 languages, 16,400 → 16,600
rows, 0 differ. C# tests 732 → 752, vitest 5,408 → 5,412.
⚠ THE GOLDEN IS INERT ON THE MODULE'S SIGNATURE. Measured over its 200 mined rows: ⟨े⟩ 1,251,
⟨ो⟩ 385, ⟨ै⟩ 170, ⟨ौ⟩ 47 — the vowel divergences are exercised hard — but ⟨॑⟩ is 9 occurrences
in 2 rows and EVERY ONE is a monosyllable, where `retainInMonosyllable` already keeps the vowel
and the reading is identical fold or no fold. ⟨ऎ ऒ ॆ ꣿ ऋ ₹ ॐ ळ⟩ are 0/200 and no golden row
carries Latin. Off-golden: the whole mined pool + the attest examples + 130 adversarial lines
= 535 lines sync and async, and 6,408 word types through `phonemizeWord`, all 0 differ.
Fixed TypeScript-first, no golden row moved:
- **`phonemizeWord` did not see the fold.** `word()` does not run the normalizer — the override
reaches `text()` only — so the path the referee eval scores read `अब॑` as *ˈəb* while the
shipped path read *ˈəbə*. The eval and the product were different engines on exactly the
construct this module exists for, and no golden could ever have shown it.
- **The fold's own comment was wrong about why it sits where it does.** It claimed the word
tokenizer "does not carry U+0951 in its class"; `DEVANAGARI_WORD` is `ऀ-ॣॲ-ॿ`, which spans
U+0951, and `phonemize("म॑थिली", "hi")` — no fold — reads one word.
- **maithili.jsonc repeated bhojpuri.jsonc's ₹ claim verbatim**, trailing half-sentence and all:
mai declares no `symbolTier`, so the shared Hindi tier claims the sign and `₹500` reads
*pˈaː̃t͡ʃ sˈəʊ ɾˈʊpje*, not the bare number.
- **⟨ꣿ⟩ U+A8FF is a declared `vowelSigns` entry the engine cannot reach.** Devanagari Extended is
outside the shared word class, so the mark ENDS the token: `कꣿ` → *kˈə* and `मꣿथिली` → two
words. Annotated rather than deleted — the alternative fix widens a class that reaches all 17
`makeNativeHindi` languages, and U+A8FF is ×0 in the corpus anyway.
Of the three findings the bho port filed against the shared Devanagari data, none recurs:
⟨ऋ⟩/⟨ृ⟩ are the TAP ɾɪ here, matching the manifest's only rhotic; ⟨अ⟩ and `inherentVowel` are both
ə; and `बजकर`/`मिनट` ARE attested for Maithili by the per-sentence morphology test. प्रतिशत is the
one word that is not, and the count is now recorded where it is used: `%` occurs 13 times across
7 of the 200 golden rows.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WuH4pRrB1PAEiG8CCLVR5S
…ents the review caught
Self-review findings, all four addressed:
- **The fold now wraps the ENGINE, not two call sites.** `createMaithili()` returns the object
`makeNativeHindi` built, and its `word`/`wordRules` were reachable unfolded — the same eval-vs-
shipped split the last commit closed on `phonemizeWord`, still open one level out.
`engine()` returns `{ ...e, word: fold∘e.word, wordRules: fold∘e.wordRules }` in TS and the
matching wrapped `NativeHindiEngine` in C#. `text` keeps the UNWRAPPED delegates by construction,
so nothing double-folds. Pinned on the public surface in both suites.
- **The C# had re-imported the stale tokenizer claim** that the same PR corrected in the TS. Gone,
along with the rest of the transcribed prose — PORTING.md gives the C# a 2-4 line header and
load-bearing notes only, and "a copy that drifts" had already drifted inside one commit.
- **`stripSymbols: "₹"` is NOT dead, and the review's reading that it was is worth pinning against.**
The shared Hindi tier claims ₹ only beside an AMOUNT, so a stranded sign still reaches the strip:
`₹ अछि` → *ˈət͡ʃʰɪ* and bare `₹` → empty, both by this entry. The manifest note now says which
inputs reach which rule, and four test rows hold it.
Gates unchanged: parity 83 languages / 16,600 rows / 0 differ, C# tests 754, vitest 5,412, and the
off-golden differential re-run clean — 535 lines sync and async, 6,408 word types. No golden moved.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WuH4pRrB1PAEiG8CCLVR5S
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WuH4pRrB1PAEiG8CCLVR5S
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Maithili runs on the shared
makeNativeHindiabugida engine, so the port is 39 lines ofdata-wiring plus one thing no sibling has: a
normalizeoverride that folds ⟨॑⟩ U+0951 ontothe avagraha ⟨ऽ⟩. 200/200 first run; the full gate goes 82 → 83 languages, 16,400 → 16,600
rows, 0 differ. C# tests 732 → 752, vitest 5,408 → 5,412.
⚠ THE GOLDEN IS INERT ON THE MODULE'S SIGNATURE. Measured over its 200 mined rows: ⟨े⟩ 1,251,
⟨ो⟩ 385, ⟨ै⟩ 170, ⟨ौ⟩ 47 — the vowel divergences are exercised hard — but ⟨॑⟩ is 9 occurrences
in 2 rows and EVERY ONE is a monosyllable, where
retainInMonosyllablealready keeps the voweland the reading is identical fold or no fold. ⟨ऎ ऒ ॆ ꣿ ऋ ₹ ॐ ळ⟩ are 0/200 and no golden row
carries Latin. Off-golden: the whole mined pool + the attest examples + 130 adversarial lines
= 535 lines sync and async, and 6,408 word types through
phonemizeWord, all 0 differ.Fixed TypeScript-first, no golden row moved:
phonemizeWorddid not see the fold.word()does not run the normalizer — the overridereaches
text()only — so the path the referee eval scores readअब॑as ˈəb while theshipped path read ˈəbə. The eval and the product were different engines on exactly the
construct this module exists for, and no golden could ever have shown it.
tokenizer "does not carry U+0951 in its class";
DEVANAGARI_WORDisऀ-ॣॲ-ॿ, which spansU+0951, and
phonemize("म॑थिली", "hi")— no fold — reads one word.mai declares no
symbolTier, so the shared Hindi tier claims the sign and₹500readspˈaː̃t͡ʃ sˈəʊ ɾˈʊpje, not the bare number.
vowelSignsentry the engine cannot reach. Devanagari Extended isoutside the shared word class, so the mark ENDS the token:
कꣿ→ kˈə andमꣿथिली→ twowords. Annotated rather than deleted — the alternative fix widens a class that reaches all 17
makeNativeHindilanguages, and U+A8FF is ×0 in the corpus anyway.Of the three findings the bho port filed against the shared Devanagari data, none recurs:
⟨ऋ⟩/⟨ृ⟩ are the TAP ɾɪ here, matching the manifest's only rhotic; ⟨अ⟩ and
inherentVowelare bothə; and
बजकर/मिनटARE attested for Maithili by the per-sentence morphology test. प्रतिशत is theone word that is not, and the count is now recorded where it is used:
%occurs 13 times across7 of the 200 golden rows.
Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01WuH4pRrB1PAEiG8CCLVR5S
Also found, NOT fixed here — the ₹ claim is FAMILY-WIDE
grep -rn 'पाँच सौ), matching' data/languages/*/*.jsoncreturns five manifests carrying theidentical wrong comment (the same text #1029 removed from bhojpuri.jsonc), including the trailing
half-sentence. Only
maithili.jsoncis fixed in this PR, because three of the others are beingported concurrently in other worktrees:
phonemize("₹500", …)maithili.jsoncpˈaː̃t͡ʃ sˈəʊ ɾˈʊpjeawadhi.jsoncpˈaː̃t͡ʃ sˈʌu ɾˈʊpjeːchhattisgarhi.jsoncpˈaː̃t͡ʃ sˈɔː ɾˈʊpjeːmagahi.jsoncpˈɑ̃t͡ʃ sˈɔ ɾˈupd͡ʒenepali.jsoncpˈãt͡s sˈʌjThat last row is the reason this is worth a table rather than a sweep: the same sentence is right in
one manifest and wrong in four.
🤖 Generated with Claude Code
https://claude.ai/code/session_01WuH4pRrB1PAEiG8CCLVR5S