Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions src/languages/macedonian/macedonian.ts
Original file line number Diff line number Diff line change
Expand Up @@ -44,12 +44,12 @@ const mkCountForm = (n: number): number => {
return m % 10 === 1 && m !== 11 ? 0 : 1;
};

// #562 symbol normalization — Macedonian. Percent/currency/units/rates carry the count form above
// symbol normalization — Macedonian. Percent/currency/units/rates carry the count form above
// (1 процент, 2 проценти; 1 километар, 2 километри). Units are written BOTH ways — Cyrillic кm and Latin
// km are equally common in the corpus — so both scripts are declared. The rate denominator is "на час"
// (per hour) but "во секунда" (per second), which the keyed unitPer expresses.
const SYMBOLS = makeSymbolNormalizer({
// #586 `multiply` — the word is this language's OWN, harvested from its existing `×` rule, so nothing new
// `multiply` — the word is this language's OWN, harvested from its existing `×` rule, so nothing new
// is sourced. Declaring it HERE is what makes ASCII `x` read like `×`: `6x6 cm` was reading the `x` as a
// LETTER NAME, and `NxN` forms outnumber `×` roughly 85 to 20 across the corpora. One word, so `by` is
// omitted and defaults to it — this language does not split dimension from product.
Expand Down Expand Up @@ -173,7 +173,7 @@ const TOKEN = /([а-шА-ШѓѕјљњќџЃЅЈЉЊЌЏѐѝЀЍ]+)|(\d+(?:,\d+)?

class MacedonianPhonemizer implements Phonemizer {
text(input: string): string {
// #562 order: Macedonian rewrites (grouping, ordinals, century/date, clock, ranges, signs) →
// order: Macedonian rewrites (grouping, ordinals, century/date, clock, ranges, signs) →
// INITIALISMS (after abbreviations, so `Д-р` is not spelled DE-ER) → the shared symbol tier last
// (it needs the number still adjacent to its unit/sign). Roman numerals arrive already converted
// at the registry seam, so regnal "Лиалофи III" is "3" by the time normalize runs.
Expand Down
31 changes: 15 additions & 16 deletions src/languages/macedonian/normalize.ts
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@
* ORDERING COUPLINGS, each a bug that happened:
* · de-grouping FIRST — a period/space is otherwise a token boundary or clause mark.
* · multi-dot era markers BEFORE the single-dot year rule, and before the `N г.` expansion.
* · the ordinal-suffix rule BEFORE the century/date rules and the range rule.
* · the ordinal-suffix rule BEFORE the century/date rules, which would otherwise claim the digits.
* · the range rule BEFORE the clock rule, so `22:00-23:00` becomes "22:00 до 23:00" first.
* · personal-initial single capitals BEFORE the initialism pass (which would otherwise see the dot).
*/
Expand Down Expand Up @@ -188,7 +188,8 @@ export function normalizeMacedonian(input: string): string {
return `${a} до ${b}`;
});

// 7) CLOCK. The colon was clause punctuation, so `06:30` read as "шест , триесет". Macedonian reads
// 6) CLOCK. The colon is otherwise clause punctuation, so `06:30` reads as "шест , триесет".
// Macedonian reads
// the time CARDINALLY with "и": 6:30 = "шест и триесет"; :00 drops the minutes (10:00 = "десет").
// The trailing "часот"/"ч" (the hour) is kept/expanded — "ч" is the abbreviation for "часот". A
// comma after the minutes marks a sports time, not a clock. The range rule ran first, so
Expand All @@ -204,12 +205,12 @@ export function normalizeMacedonian(input: string): string {
return typeof ч === "string" ? `${body} часот` : body;
},
);
// 7b) `am`/`pm` after a time → претпладне/попладне. Standalone words (the clock already consumed the
// 6b) `am`/`pm` after a time → претпладне/попладне. Standalone words (the clock already consumed the
// digits), matched on word boundaries so "Сами" is untouched.
s = s.replace(/\bam\b/giu, "претпладне");
s = s.replace(/\bpm\b/giu, "попладне");

// 8) THE ORDINAL SUFFIXthe big one. `(\d+)[- ]?` + the written suffix (the last letters of the
// 7) THE ORDINAL SUFFIX, the largest class in this file. `(\d+)[- ]?` + the written suffix (the last letters of the
// spoken ordinal). The `-те` suffix is ambiguous between a count ("the N") and a decade: a 4-digit
// number followed by "години" is a decade, otherwise it is "the N".
s = s.replace(
Expand Down Expand Up @@ -241,9 +242,9 @@ export function normalizeMacedonian(input: string): string {
},
);

// 9) CENTURY — `N век` → ordinal + век (10 век → десетти век), including the compound list
// 8) CENTURY — `N век` → ordinal + век (10 век → десетти век), including the compound list
// `10 и 11 век`. Also the one Germanic remnant `8. век` (with a dot). The suffix forms (17-ти век,
// 18-тиот век) were already claimed by step 8.
// 18-тиот век) are already claimed by step 7.
s = s.replace(/(\d+)\s+и\s+(\d+)\s*век(?![\p{L}\p{M}])/gu, (_m, a: string, b: string) => {
const oa = mkOrdinal(Number(a)), ob = mkOrdinal(Number(b));
return oa !== undefined && ob !== undefined ? `${oa} и ${ob} век` : _m;
Expand All @@ -257,7 +258,7 @@ export function normalizeMacedonian(input: string): string {
return o === undefined ? m0 : `${o} век`;
});

// 10) DATES — `N месяц` → ordinal + месяц (на 6 октомври → на шести октомври), including a date range
// 9) DATES — `N <month>` → ordinal + month (на 6 октомври → на шести октомври), including a date range
// `24 август - 5 септември` and the Germanic-dot form `4. јули 1776`. The day must be 1–31.
const monthAlt = [...MONTHS].join("|");
s = s.replace(
Expand All @@ -277,9 +278,7 @@ export function normalizeMacedonian(input: string): string {
},
);

// 11) (the range rule lives at step 5, before the clock.)

// 11) REGNAL ORDINALS. All three corpus Romans (Лиалофи III, Елизабета II, Луј XVI) are regnal proper
// 10) REGNAL ORDINALS. All three corpus Romans (Лиалофи III, Елизабета II, Луј XVI) are regnal proper
// names, and the shared Roman pass has already rewritten them to CARDINAL digits before this engine
// runs (Macedonian has no ROMAN_POLICIES entry). The digit after a capitalized NAME is read as an
// ordinal: Лиалофи 3 → Лиалофи трети, Луј 16 → Луј шеснаесетти. Guarded by the same ≤39 monarch
Expand All @@ -297,7 +296,7 @@ export function normalizeMacedonian(input: string): string {
return `${name} ${name.endsWith("а") ? femIndef(o) : o}`;
});

// 12) RATE UNITS the shared tier cannot compose. `милји/час` (miles/hour — милји is a full word, not an
// 11) RATE UNITS the shared tier cannot compose. `милји/час` (miles/hour — милји is a full word, not an
// abbreviation), the Latin `mph`/`kph`, `Mbit/s` (megabits per second). The Cyrillic squared units
// `мм2`/`км2` are also local: the tier's exponent lookbehind `(?<=[a-zA-Z])` is ASCII-only and the
// corpus writes `3136 мм2`.
Expand All @@ -308,15 +307,15 @@ export function normalizeMacedonian(input: string): string {
s = s.replace(/(\d+)\s*мм\s*[²2](?!\d)/gu, "$1 квадратни милиметри");
s = s.replace(/(\d+)\s*км\s*[²2](?!\d)/gu, "$1 квадратни километри");

// 13) DEGREES — `90°F`, `35° W`, and a bare `N°`. The corpus's own spelled-out form is
// 12) DEGREES — `90°F`, `35° W`, and a bare `N°`. The corpus's own spelled-out form is
// "30 степени целзиусови", so °C/°F use the "по" construction; `° W`/`° E` are coordinates.
s = s.replace(/(\d+)\s*°\s*C(?![\p{L}\p{M}])/gu, "$1 степени по Целзиус");
s = s.replace(/(\d+)\s*°\s*F(?![\p{L}\p{M}])/gu, "$1 степени по Фаренхаjт");
s = s.replace(/(\d+)\s*°\s*W(?![\p{L}\p{M}])/gu, "$1 степени запад");
s = s.replace(/(\d+)\s*°\s*E(?![\p{L}\p{M}])/gu, "$1 степени исток");
s = s.replace(/(\d+)\s*°/gu, "$1 степени");

// 14) SIGNS. `+30` (the corpus's "над +30 степени") reads "плус". Minus, ×, ÷, =, <, > and the
// 13) SIGNS. `+30` (the corpus's "над +30 степени") reads "плус". Minus, ×, ÷, =, <, > and the
// ampersand are the handoff's sign classes — none occurs in the corpus but a dropped sign is
// inaudible, so each is read.
s = s.replace(/(^|[\s(])[-−]\s?(?=\d)/gu, "$1минус ");
Expand All @@ -334,12 +333,12 @@ export function normalizeMacedonian(input: string): string {
s = s.replace(/\s*>\s*/gu, " поголемо од ");
s = s.replace(/\s*[&&]\s*/gu, " и ");

// 15) pH → "пе ха" (letter names) and `Ghz` → "гигахерци" — the corpus's lowercase-tech tokens that
// 14) pH → "пе ха" (letter names) and `Ghz` → "гигахерци" — the corpus's lowercase-tech tokens that
// would otherwise read as consonant clusters or Latin foreign.
s = s.replace(/(?<![\p{L}\p{M}])pH(?![\p{L}\p{M}])/gu, "пе ха");
s = s.replace(/(?<![\p{L}\p{M}])Ghz(?![\p{L}\p{M}])/giu, "гигахерци");

// 15b) FRACTIONS — the corpus's "29¾ инчи на 24½ инчи" and "5 мм (1/5 инчи)". ¾/½ after a whole read
// 14b) FRACTIONS — the corpus's "29¾ инчи на 24½ инчи" and "5 мм (1/5 инчи)". ¾/½ after a whole read
// "и три четвртини" / "и половина"; a unit fraction reads "една петтина"-shaped (numeral-ordinand
// stem + -тина). The vulgar-fraction glyphs are not in any clause-punctuation map, so they were
// being dropped outright.
Expand All @@ -354,7 +353,7 @@ export function normalizeMacedonian(input: string): string {
return `една ${suffix}`;
});

// 16) PERSONAL-INITIAL single capitals — `Н. Вејн` (the shared LONE_INITIAL handles `В. Буш`, which
// 15) PERSONAL-INITIAL single capitals — `Н. Вејн` (the shared LONE_INITIAL handles `В. Буш`, which
// sits between capitalized words; this one follows a comma in "НАСА, Н. Вејн"). And a lone Latin
// capital standing as a LETTER between words — `буквата V` (the letter vee) reads "ве" not the
// English [viː]; `H во pH` (after pH → "пе ха") reads "ха".
Expand Down
8 changes: 4 additions & 4 deletions src/languages/madurese/madurese.ts
Original file line number Diff line number Diff line change
Expand Up @@ -137,10 +137,10 @@ export function phonemizeWord(word: string): string {
const TOKEN = new RegExp(`(${hostWordRun(["Latin"], "'")})|(\\d+)|([.!?…,;:])`, "giu");

/**
* This language's OWN inventory — the TOKEN word class as it stood before the widening above, lifted
* verbatim, so nothing about the orthography is invented here. A token this REJECTS carries a letter the
* language does not use, i.e. a foreign name. See core/hostWord.ts: this is the INVENTORY question, and it
* is no longer also deciding where the script boundary falls.
* This language's OWN inventory. ⚠ TWO DIFFERENT QUESTIONS, KEPT APART: the TOKEN class above decides where
* the SCRIPT boundary falls (routing), while this one decides whether the g2p has rules for these letters. A
* token this class REJECTS carries a letter the language does not usei.e. a foreign name. See
* core/hostWord.ts.
*/
const NATIVE_CLASS = "[a-zâèéòḍṭ']";
const nat = makeNativiser(NATIVE_CLASS, "iu");
Expand Down
35 changes: 18 additions & 17 deletions src/languages/magahi/magahi.ts
Original file line number Diff line number Diff line change
@@ -1,24 +1,25 @@
/**
* Native Magahi / मगही (mag) text phonemizer — canonical IPA. Indo-Aryan (Bihari, Magadhan),
* Devanagari. Reuses the Hindi/Bhojpuri Devanagari engine (makeNativeHindi — schwa deletion, weight stress) with a
* Magahi data file. Magahi shares the Bihari core with Bhojpuri — NO phonemic vowel length, single sibilant श/ष→[s],
* ण/ञ→[n] — but differs in its documented GLIDE HARDENING (Vinod Kumar 2026, A Comparative Phonological Study of
* Bihari Languages, §6.2): word-initial व→[b] (वंश→bans) and य→[d͡ʒ] (यन्त्र→jantar), where Bhojpuri preserves the
* glides (व→w, य→j). Single comparative-source delta on the grammar-anchored Bhojpuri base → 🔷. Was a mag→bho
* alias until the reference revealed the delta.
* Native Magahi / मगही (mag) text phonemizer — canonical IPA. Indo-Aryan (Bihari, Magadhan), Devanagari.
* Reuses the Hindi/Bhojpuri Devanagari engine (makeNativeHindi — schwa deletion, weight stress) with a Magahi
* data file.
*
* Magahi shares the Bihari core with Bhojpuri — NO phonemic vowel length, single sibilant श/ष→[s], ण/ञ→[n] —
* but differs in its documented GLIDE HARDENING (Vinod Kumar 2026, *A Comparative Phonological Study of Bihari
* Languages*, §6.2): word-initial व→[b] (वंश→bans) and य→[d͡ʒ] (यन्त्र→jantar), where Bhojpuri preserves the
* glides (व→w, य→j). ⚠ That delta rests on a SINGLE comparative source over the grammar-anchored Bhojpuri base.
*/
/**
* NORMALIZER WORDS: the Hindi defaults are RETAINED and four are confirmed for Magahi, one of them by a
* DEFINITION rather than a usage. Evidence from mag.wikipedia via `tools/normalization/attest.ts`
* (`tools/corpus/attest/mag.jsonc`); sentences judged Magahi by हे / हल / हलै / आउ / -के:
* NORMALIZER WORDS. This engine inherits Hindi's, and four are confirmed for Magahi — sentences judged
* Magahi by हे / हल / हलै / आउ / -के:
*
* प्रतिशत a DEFINITIONAL citation, which outranks any usage example: "प्रतिशतके अर्थ हे प्रति सौ या
* प्रति सैकड़ा (% = 1/100)" ties the word directly to the SIGN this layer is reading.
* बजे "रातके १२ बजे होलै हल"
* मिनट "प्रति मिनट क्रान्ति, सङ्क्षिप्त आर/मिनट या आरपिएम हे"
* ईसा पूर्व "ईसा पूर्व ३०४ से ईसा पूर्व २३२ … हल"
*
* प्रतिशत ✓✓ the Magahi wiki's own article DEFINES it: "प्रतिशतके अर्थ हे प्रति सौ या प्रति सैकड़ा
* (% = 1/100)" — a definitional citation tying the word to the SIGN, which is stronger than
* any usage example. Also in use: "साक्षरता दर ७५ प्रतिशत … हलै आउ".
* बजे ✓ "रातके १२ बजे होलै हल"
* मिनट ✓ "प्रति मिनट क्रान्ति, सङ्क्षिप्त आर/मिनट या आरपिएम हे"
* ईसा पूर्व ✓ "ईसा पूर्व ३०४ से ईसा पूर्व २३२ … हल"
* बजकर NOT ATTESTED (0 hits) — Hindi's clock connective stands, unconfirmed.
* बजकर, Hindi's clock connective, is NOT attested for Magahi. It stands unconfirmed rather than being
* replaced — an unsourced substitute is worse than an inherited word.
*/
import { makeNativeHindi, type HindiDef, type ForeignPhonemizer } from "../hindi/hindi.ts";
import { loadManifest } from "../../core/loadManifest.ts";
Expand Down
35 changes: 17 additions & 18 deletions src/languages/maithili/maithili.ts
Original file line number Diff line number Diff line change
@@ -1,26 +1,25 @@
/**
* Native Maithili / मैथिली (mai) text phonemizer — canonical IPA. Eastern Indo-Aryan (Bihari
* group), Devanagari. Reuses the Hindi Devanagari engine (makeNativeHindi — schwa deletion, weight stress,
* numbers) with a Maithili data file whose divergences from Hindi are: SHORT e/o (ए→e, ओ→o), the diphthongs
* ऐ→[əɪ] / औ→[əu], and inherent /ə/. Maithili's signature — a cluster schwa that Hindi DELETES instead reduces to
* an ULTRASHORT [ᵊ] (इसपात→ɪsᵊpaːt) — is a narrow phonetic detail folded (ᵊ~∅) against the referee.
* Native Maithili / मैथिली (mai) text phonemizer — canonical IPA. Eastern Indo-Aryan (Bihari group),
* Devanagari. Reuses the Hindi Devanagari engine (makeNativeHindi — schwa deletion, weight stress, numbers)
* with a Maithili data file. The divergences from Hindi: SHORT e/o (ए→e, ओ→o), the diphthongs ऐ→[əɪ] /
* औ→[əu], and inherent /ə/.
*
* 🔷 SINGLE-SOURCE: the only referee is wikipron mai_deva narrow (167 human pairs) — small, so this is a
* single-source-verified bring-up, not a confident convergence. Distinct from the ⛔ Bhojpuri/Awadhi stubs, which
* have NO referee at all.
* Maithili's signature — a cluster schwa that Hindi DELETES is instead reduced to an ULTRASHORT [ᵊ]
* (इसपात→ɪsᵊpaːt) — is a narrow phonetic detail, folded (ᵊ~∅) rather than contrasted.
*
* ⚠ SINGLE-SOURCE bring-up: the only referee is a small human set (167 pairs), so this is verified rather
* than convergent.
*/
/**
* NORMALIZER WORDS: Hindi's CLOCK words are confirmed for Maithili; its percent word is NOT, and is
* retained unconfirmed. Evidence from mai.wikipedia via `tools/normalization/attest.ts`
* (`tools/corpus/attest/mai.jsonc`). This language is why the reading has to be done per sentence:
* NORMALIZER WORDS. This engine inherits Hindi's. The CLOCK words (बजकर, मिनट) are confirmed for Maithili;
* the percent word प्रतिशत is NOT, and is retained unconfirmed — an unsourced substitute is worse than an
* inherited word.
*
* बजकर + मिनट ✓ "प्राण प्रतिष्ठाक मुहूर्त १२ बजकर २९ मिनट ८ सेकेण्ड सँ … रहल जे"
* — प्रतिष्ठाक / सेकेण्ड सँ / रहल जे are Maithili. TWO of the four बजकर hits are Maithili;
* the other two are the same passage in HINDI (का मुहूर्त, रहा, मिलाकर), quoted verbatim.
* मिनट ✓ "5 घंटा 30 मिनट जोड़ला सं प्राप्त होइत अछि" — होइत अछि is Maithili
* प्रतिशत ✗ its single hit is a NEPALI passage ("९७.९ प्रतिशत भन्दा अधिक … लागेकाछन्"), so it says
* nothing about Maithili. Hindi's default stands, unconfirmed — not replaced, because an
* unsourced substitute is worse than an inherited word (this issue's own standing rule).
* ⚠ ATTESTATION MUST BE JUDGED PER SENTENCE, NOT PER HIT, and this language is why. Devanagari wikis quote
* each other verbatim, so a search for a Maithili word returns Hindi and Nepali passages under a Maithili
* title: half the बजकर hits here are one Hindi passage quoted whole (का मुहूर्त, रहा, मिलाकर), and the
* only प्रतिशत hit is Nepali (भन्दा अधिक, लागेकाछन्). A hit counts only when the surrounding sentence
* carries Maithili morphology — होइत अछि, रहल जे, सेकेण्ड सँ.
*/
import { makeNativeHindi, type HindiDef, type ForeignPhonemizer } from "../hindi/hindi.ts";
import { loadManifest } from "../../core/loadManifest.ts";
Expand Down
Loading
Loading