Skip to content

mai: port Maithili to C#, and pin the divergences the golden cannot see - #1033

Merged
christopherthompson81 merged 3 commits into
mainfrom
port/maithili-csharp
Aug 26, 2026
Merged

mai: port Maithili to C#, and pin the divergences the golden cannot see#1033
christopherthompson81 merged 3 commits into
mainfrom
port/maithili-csharp

Conversation

@christopherthompson81

Copy link
Copy Markdown
Owner

Maithili runs on the shared makeNativeHindi abugida engine, so the port is 39 lines of
data-wiring plus one thing no sibling has: a normalize override that folds ⟨॑⟩ U+0951 onto
the avagraha ⟨ऽ⟩. 200/200 first run; the full gate goes 82 → 83 languages, 16,400 → 16,600
rows, 0 differ. C# tests 732 → 752, vitest 5,408 → 5,412.

⚠ THE GOLDEN IS INERT ON THE MODULE'S SIGNATURE. Measured over its 200 mined rows: ⟨े⟩ 1,251,
⟨ो⟩ 385, ⟨ै⟩ 170, ⟨ौ⟩ 47 — the vowel divergences are exercised hard — but ⟨॑⟩ is 9 occurrences
in 2 rows and EVERY ONE is a monosyllable, where retainInMonosyllable already keeps the vowel
and the reading is identical fold or no fold. ⟨ऎ ऒ ॆ ꣿ ऋ ₹ ॐ ळ⟩ are 0/200 and no golden row
carries Latin. Off-golden: the whole mined pool + the attest examples + 130 adversarial lines
= 535 lines sync and async, and 6,408 word types through phonemizeWord, all 0 differ.

Fixed TypeScript-first, no golden row moved:

  • phonemizeWord did not see the fold. word() does not run the normalizer — the override
    reaches text() only — so the path the referee eval scores read अब॑ as ˈəb while the
    shipped path read ˈəbə. The eval and the product were different engines on exactly the
    construct this module exists for, and no golden could ever have shown it.
  • The fold's own comment was wrong about why it sits where it does. It claimed the word
    tokenizer "does not carry U+0951 in its class"; DEVANAGARI_WORD is ऀ-ॣॲ-ॿ, which spans
    U+0951, and phonemize("म॑थिली", "hi") — no fold — reads one word.
  • maithili.jsonc repeated bhojpuri.jsonc's ₹ claim verbatim, trailing half-sentence and all:
    mai declares no symbolTier, so the shared Hindi tier claims the sign and ₹500 reads
    pˈaː̃t͡ʃ sˈəʊ ɾˈʊpje, not the bare number.
  • ⟨ꣿ⟩ U+A8FF is a declared vowelSigns entry the engine cannot reach. Devanagari Extended is
    outside the shared word class, so the mark ENDS the token: कꣿkˈə and मꣿथिली → two
    words. Annotated rather than deleted — the alternative fix widens a class that reaches all 17
    makeNativeHindi languages, and U+A8FF is ×0 in the corpus anyway.

Of the three findings the bho port filed against the shared Devanagari data, none recurs:
⟨ऋ⟩/⟨ृ⟩ are the TAP ɾɪ here, matching the manifest's only rhotic; ⟨अ⟩ and inherentVowel are both
ə; and बजकर/मिनट ARE attested for Maithili by the per-sentence morphology test. प्रतिशत is the
one word that is not, and the count is now recorded where it is used: % occurs 13 times across
7 of the 200 golden rows.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01WuH4pRrB1PAEiG8CCLVR5S

Also found, NOT fixed here — the ₹ claim is FAMILY-WIDE

grep -rn 'पाँच सौ), matching' data/languages/*/*.jsonc returns five manifests carrying the
identical wrong comment (the same text #1029 removed from bhojpuri.jsonc), including the trailing
half-sentence. Only maithili.jsonc is fixed in this PR, because three of the others are being
ported concurrently in other worktrees:

manifest line phonemize("₹500", …) verdict
maithili.jsonc 303 pˈaː̃t͡ʃ sˈəʊ ɾˈʊpje false — fixed here
awadhi.jsonc 313 pˈaː̃t͡ʃ sˈʌu ɾˈʊpjeː false
chhattisgarhi.jsonc 296 pˈaː̃t͡ʃ sˈɔː ɾˈʊpjeː false
magahi.jsonc 311 pˈɑ̃t͡ʃ sˈɔ ɾˈupd͡ʒe false (nobody is porting mag)
nepali.jsonc 194 pˈãt͡s sˈʌj TRUE — ne really does read the bare number, so its copy of the comment is correct and must be left alone

That last row is the reason this is worth a table rather than a sweep: the same sentence is right in
one manifest and wrong in four.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WuH4pRrB1PAEiG8CCLVR5S

christopherthompson81 and others added 3 commits August 25, 2026 23:44
Maithili runs on the shared `makeNativeHindi` abugida engine, so the port is 39 lines of
data-wiring plus one thing no sibling has: a `normalize` override that folds ⟨॑⟩ U+0951 onto
the avagraha ⟨ऽ⟩. 200/200 first run; the full gate goes 82 → 83 languages, 16,400 → 16,600
rows, 0 differ. C# tests 732 → 752, vitest 5,408 → 5,412.

⚠ THE GOLDEN IS INERT ON THE MODULE'S SIGNATURE. Measured over its 200 mined rows: ⟨े⟩ 1,251,
⟨ो⟩ 385, ⟨ै⟩ 170, ⟨ौ⟩ 47 — the vowel divergences are exercised hard — but ⟨॑⟩ is 9 occurrences
in 2 rows and EVERY ONE is a monosyllable, where `retainInMonosyllable` already keeps the vowel
and the reading is identical fold or no fold. ⟨ऎ ऒ ॆ ꣿ ऋ ₹ ॐ ळ⟩ are 0/200 and no golden row
carries Latin. Off-golden: the whole mined pool + the attest examples + 130 adversarial lines
= 535 lines sync and async, and 6,408 word types through `phonemizeWord`, all 0 differ.

Fixed TypeScript-first, no golden row moved:

- **`phonemizeWord` did not see the fold.** `word()` does not run the normalizer — the override
  reaches `text()` only — so the path the referee eval scores read `अब॑` as *ˈəb* while the
  shipped path read *ˈəbə*. The eval and the product were different engines on exactly the
  construct this module exists for, and no golden could ever have shown it.
- **The fold's own comment was wrong about why it sits where it does.** It claimed the word
  tokenizer "does not carry U+0951 in its class"; `DEVANAGARI_WORD` is `ऀ-ॣॲ-ॿ`, which spans
  U+0951, and `phonemize("म॑थिली", "hi")` — no fold — reads one word.
- **maithili.jsonc repeated bhojpuri.jsonc's ₹ claim verbatim**, trailing half-sentence and all:
  mai declares no `symbolTier`, so the shared Hindi tier claims the sign and `₹500` reads
  *pˈaː̃t͡ʃ sˈəʊ ɾˈʊpje*, not the bare number.
- **⟨ꣿ⟩ U+A8FF is a declared `vowelSigns` entry the engine cannot reach.** Devanagari Extended is
  outside the shared word class, so the mark ENDS the token: `कꣿ` → *kˈə* and `मꣿथिली` → two
  words. Annotated rather than deleted — the alternative fix widens a class that reaches all 17
  `makeNativeHindi` languages, and U+A8FF is ×0 in the corpus anyway.

Of the three findings the bho port filed against the shared Devanagari data, none recurs:
⟨ऋ⟩/⟨ृ⟩ are the TAP ɾɪ here, matching the manifest's only rhotic; ⟨अ⟩ and `inherentVowel` are both
ə; and `बजकर`/`मिनट` ARE attested for Maithili by the per-sentence morphology test. प्रतिशत is the
one word that is not, and the count is now recorded where it is used: `%` occurs 13 times across
7 of the 200 golden rows.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WuH4pRrB1PAEiG8CCLVR5S
…ents the review caught

Self-review findings, all four addressed:

- **The fold now wraps the ENGINE, not two call sites.** `createMaithili()` returns the object
  `makeNativeHindi` built, and its `word`/`wordRules` were reachable unfolded — the same eval-vs-
  shipped split the last commit closed on `phonemizeWord`, still open one level out.
  `engine()` returns `{ ...e, word: fold∘e.word, wordRules: fold∘e.wordRules }` in TS and the
  matching wrapped `NativeHindiEngine` in C#. `text` keeps the UNWRAPPED delegates by construction,
  so nothing double-folds. Pinned on the public surface in both suites.
- **The C# had re-imported the stale tokenizer claim** that the same PR corrected in the TS. Gone,
  along with the rest of the transcribed prose — PORTING.md gives the C# a 2-4 line header and
  load-bearing notes only, and "a copy that drifts" had already drifted inside one commit.
- **`stripSymbols: "₹"` is NOT dead, and the review's reading that it was is worth pinning against.**
  The shared Hindi tier claims ₹ only beside an AMOUNT, so a stranded sign still reaches the strip:
  `₹ अछि` → *ˈət͡ʃʰɪ* and bare `₹` → empty, both by this entry. The manifest note now says which
  inputs reach which rule, and four test rows hold it.

Gates unchanged: parity 83 languages / 16,600 rows / 0 differ, C# tests 754, vitest 5,412, and the
off-golden differential re-run clean — 535 lines sync and async, 6,408 word types. No golden moved.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WuH4pRrB1PAEiG8CCLVR5S
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant