core/ipa: ⟨ɑ ɒ ʏ⟩ join the nucleus class — stress was vanishing, bhojpuri +125 words - #752
Merged
Merged
Conversation
…bhojpuri gains 125 words I deferred this twice on the grounds that the referees carry no stress marks, so placement could not be adjudicated. Both times I asked only which placement was right, and never asked what happens to a word with NO visible nucleus. The answer is that weightStress emits nothing at all: آپ 'you' → ɑːp, کام 'work' → kɑːm, بازار → bɑːzɑːɾ, all bare, while the minimal pair kaːm → kˈaːm was stressed all along. No referee is needed to call that wrong. And the referee then adjudicated anyway, through the other language: ur is unchanged on both its corpora (4382/7709 and 3364/5667 — genuinely stress-folded), while bho goes 1008 → 1133/1623, 62.1% → 69.8%. IPA_VOWELS is not only the stress nucleus; the Indic engines build segmental regexes from it too, so an invisible ⟨ɑ⟩ was corrupting more than stress. 18 goldens updated. Eight are stress APPEARING where there was none (آباد ہاتھ آواز نام ہزار مواد مربع 60واں); the rest are placement moving to what the documented weight rule gives once it can see every syllable. A test now pins the bare-word case directly. IPA_VOWELS and IPA_VOWEL become one list again — the split in #748 was a workaround for a question that had an answer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Lc3WnUgogC7okV7n53vjr
The bhojpuri gain is 135 fixed / 10 broken, and the 10 are one class: words written with ⟨ऽ⟩ (U+0973), which appears in neither engine and is dropped, while the invisible ⟨ɑ⟩ had been accidentally suppressing their final-schwa deletion. Removing the accident exposed a real gap rather than creating one. Logged with its scope (31 bho + 2 mai words, nothing else in any referee) as its own change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Lc3WnUgogC7okV7n53vjr
christopherthompson81
force-pushed
the
ipa-alpha-nucleus
branch
from
August 8, 2026 05:23
8d7899d to
529421f
Compare
christopherthompson81
added a commit
that referenced
this pull request
Aug 8, 2026
…1133 → 1153 The residual under #752's +125: ⟨ऽ⟩ (U+093D — not U+0973 as reported; I dumped the codepoints) is unknown to the engine and dropped, and the invisible ⟨ɑ⟩ had been accidentally suppressing final-schwa deletion for these forms. Removing that accident exposed the real gap. In Bhojpuri the avagraha is a LIVE orthographic mark on the imperative/participial forms, not Sanskrit's elision sign: it writes exactly the inherent vowel the schwa rule removes, so कर/करऽ is a minimal pair. All 31 forms in the grammar-mined referee keep the vowel and none delete it. A retainOnAvagraha flag on the manifest's schwaDeletion block, tested against the SPELLING — g2p drops the character, so nothing survives into the phones to condition on. That makes it the only retain-condition not decidable from the IPA string, which the comment now says. bho 1133 → 1153/1623 (69.8% → 71.0%), mai 141 → 142/167. Opt-in, so the other 15 engines composing makeNativeHindi are untouched — hi/mr/ne/awa re-measured identical to main. Deliberate residual: Maithili's referee transcribes the retained vowel LONG and we emit short. Retention is right in both; the length is two words' worth of Maithili detail. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Lc3WnUgogC7okV7n53vjr
christopherthompson81
added a commit
that referenced
this pull request
Aug 8, 2026
…1133 → 1153 (#754) * bhojpuri/maithili: a word-final avagraha ⟨ऽ⟩ retains the vowel — bho 1133 → 1153 The residual under #752's +125: ⟨ऽ⟩ (U+093D — not U+0973 as reported; I dumped the codepoints) is unknown to the engine and dropped, and the invisible ⟨ɑ⟩ had been accidentally suppressing final-schwa deletion for these forms. Removing that accident exposed the real gap. In Bhojpuri the avagraha is a LIVE orthographic mark on the imperative/participial forms, not Sanskrit's elision sign: it writes exactly the inherent vowel the schwa rule removes, so कर/करऽ is a minimal pair. All 31 forms in the grammar-mined referee keep the vowel and none delete it. A retainOnAvagraha flag on the manifest's schwaDeletion block, tested against the SPELLING — g2p drops the character, so nothing survives into the phones to condition on. That makes it the only retain-condition not decidable from the IPA string, which the comment now says. bho 1133 → 1153/1623 (69.8% → 71.0%), mai 141 → 142/167. Opt-in, so the other 15 engines composing makeNativeHindi are untouched — hi/mr/ne/awa re-measured identical to main. Deliberate residual: Maithili's referee transcribes the retained vowel LONG and we emit short. Retention is right in both; the length is two words' worth of Maithili detail. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Lc3WnUgogC7okV7n53vjr * review: the avagraha sign is per-SCRIPT, and maithili had no test behind its flag Three fixes, the middle one substantive: - /\u093D$/ hard-coded the DEVANAGARI avagraha while makeNativeHindi is script-parameterized and gujarati already passes its own block. Any non-Devanagari language setting the flag would have got a manifest that reads as enabled while every final schwa still deleted. The sign now lives on AbugidaScript (Brahmic blocks are aligned — offset 0x3D in each: ऽ U+093D, ঽ U+09BD, ઽ U+0ABD), and a manifest asking for retention under a script that declares none THROWS at construction instead of doing nothing quietly. Verified the throw fires rather than assuming it. - maithili opted in with no rationale and nothing asserting it; a later re-tune of its schwa block could have flipped it back silently. It now carries the reason, the thin evidence (two referee forms), and goldens — including the known length divergence, pinned to what the engine does rather than what the referee wants. - the Run 10 heading was dated ahead of the commit it records. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Lc3WnUgogC7okV7n53vjr --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The last residual from #748 — the one I twice said was blocked. It wasn't, and both the diagnosis and the evidence turned out to be available all along.
What I kept missing
I deferred this on the grounds that the ur/bho referees carry no stress marks, so stress PLACEMENT can't be adjudicated. True — but placement was never the only thing at stake. I never asked what happens to a word with no visible nucleus at all:
The weight rule's fallback chain ends at "first syllable" — but with no nucleus visible there are no syllables, so it emits nothing. In the live engine:
آپ"you" →ɑːp,کام"work" →kɑːm,بازار→bɑːzɑːɾ. Everyday words, silently unstressed. No referee is needed to call that wrong.And the referee did adjudicate — through the other language
ur.wikipron-urd-broadur.cle-speechbho.grammar-mined+125 words. Urdu really is stress-folded and can't see the change; Bhojpuri can, because
IPA_VOWELSis not only the stress nucleus — the Indic engines build SEGMENTAL regexes from it too, so an invisible ⟨ɑ⟩ was corrupting more than stress. The evidence I said didn't exist was one command away, in the corpus I'd already been told was in-repo.The golden changes
18 updated, and they split:
ɑːbɑːd̪→ɑːbˈɑːd̪,ɦɑːt̪ʰ→ɦˈɑːt̪ʰ,ɑːʋɑːz→ɑːʋˈɑːz,nɑːm→nˈɑːm,ɦzɑːɾ→ɦzˈɑːɾ,mʋɑːd̪→mʋˈɑːd̪,sɑːʈʰʋɑː̃→sˈɑːʈʰʋɑː̃,mʊɾəbbˈɑːʔ.A new test pins the bare-word case directly, so the actual bug can't come back quietly.
Also
IPA_VOWELSandIPA_VOWELcollapse back into one list. The split #748 introduced was a workaround for a question that had an answer.npm run cigreen — 228 files / 3137 tests.docs/ipa_classes_investigation.mdRun 9 records the miss: I scoped the question to the first thing I noticed and treated one corpus's silence as the whole evidence base.🤖 Generated with Claude Code
https://claude.ai/code/session_012Lc3WnUgogC7okV7n53vjr
Review addendum
Independent measurement confirmed the blast radius:
gu/mr/pa/hichange 0 words,sdchanges 18 (stress-only),urchanges 2550 — all stress-only, zero segmental, so the claim that the Urdu referees can't see it is exact. Every referee score is identical except bho.The bho +125 breaks down as 135 fixed / 10 broken, and the 10 are one class: words written with the avagraha ⟨ऽ⟩ (U+0973), which appears in neither the Bhojpuri nor Hindi engine and is dropped — the invisible ⟨ɑ⟩ had been accidentally suppressing their final-schwa deletion. Removing the accident exposed a pre-existing gap rather than creating one (avagraha matches 15/31 → 5/31). Scope is 31 bho + 2 mai words; nothing else in any referee carries the character. Recorded in the investigation doc and left as its own change, since
schwaDeletionin the manifest is already the hook for it.Also confirmed: no residual "no stress at all" outputs remain except genuinely vowelless tokens (
ء→ʔ,ળ→ɭ), andIPA_VOWELSis now the complete IPA vowel chart with no regex metacharacters, so[${IPA_VOWELS}]stays safe to interpolate.