Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 4 additions & 5 deletions src/languages/afrikaans/normalize.ts
Original file line number Diff line number Diff line change
Expand Up @@ -278,11 +278,10 @@ export function normalizeAfrikaans(input: string): string {
// in this corpus is a RANGE or SCORE (`2-3 km`, `7-2`, `1469-1539`, `35-40mph`), so the rule only
// fires when the minus is NOT between two digits (a leading negative) — exactly the DROP-test shape.
// `=`, `<`, `>`, `×`, `÷` do not occur in af_za but are read for completeness.
// ⚠ ± IS THIS LANGUAGE'S OWN TWO WORDS, juxtaposed — zero new sourcing. Both halves are lifted from
// the plus and minus rules in this file, so nothing is invented, and both are SIGN names rather than
// operation names, which is what ± needs: it marks a tolerance, not an addition. The FORM is the one every
// language that already read ± uses (bg/da/is/nb/ro/sv juxtapose with no conjunction). Runs BEFORE the +
// rule, since ± is a single character the + rule cannot see.
// ⚠ ± IS A SINGLE CHARACTER (U+00B1), NOT A `+`, so no `+` rule can ever match inside it. It needs
// its own rule or the sign is dropped in silence; ordering against the `+` rule is free. The
// reading is this language's own two words juxtaposed, and ⚠ both are SIGN names rather than
// OPERATION names, which is what ± needs: it marks a TOLERANCE, not an addition.
s = s.replace(/±/gu, " plus of minus ");
s = s.replace(/\+\s?(?=\d)/gu, " plus ");
s = s.replace(/(?<![\p{L}\p{Nd}])-(\d+)(?!\s*[-\d])/gu, "minus $1");
Expand Down
6 changes: 3 additions & 3 deletions src/languages/arabic/normalize.ts
Original file line number Diff line number Diff line change
Expand Up @@ -78,9 +78,9 @@ export function normalizeArabic(input: string): string {
// 5) SIGNS. A dropped sign is silent content loss, so these are read whether or not a given corpus has
// an instance.
s = s.replace(new RegExp(`(^|[\\s(])[-−–]([${DIGIT}])`, "gu"), "$1نَاقِص $2");
// ⚠ ± IS THE TWO SIGN WORDS JUXTAPOSED, with no conjunction — the form bg/da/is/nb/ro/sv all use, where
// English is the outlier that needs "or". ⚠ RUNS BEFORE THE `+` RULE: ± is a single character the `+`
// rule cannot see, so putting it second would leave it unread.
// ⚠ ± IS THE TWO SIGN WORDS JUXTAPOSED, with no conjunction. It needs its OWN rule: ± is a single
// character (U+00B1), not a `+`, so no `+` rule can match inside it and the sign would otherwise
// be dropped in silence.
s = s.replace(/±/gu, " زَائِد نَاقِص ");
s = s.replace(new RegExp(`(\\S)\\+\\s?([${DIGIT}])`, "gu"), "$1 زَائِد $2");
s = s.replace(new RegExp(`(^|\\s)\\+\\s?([${DIGIT}])`, "gu"), "$1زَائِد $2");
Expand Down
10 changes: 4 additions & 6 deletions src/languages/aromanian/aromanian.ts
Original file line number Diff line number Diff line change
Expand Up @@ -105,12 +105,10 @@ const TOKEN = new RegExp(`(${LATIN_RUN})|(\\d+)|([.?!,;:…])`, "gu");
* language does not use, i.e. a foreign name. See core/hostWord.ts: this is the INVENTORY question, and it
* is no longer also deciding where the script boundary falls.
*
* ⚠ NOT QUITE VERBATIM: ä Ä were REMOVED, because the g2p has no rule for them and DROPPED them outright.
* The old token class listed them anyway, and the word-level fold hid the mismatch — a word containing one was
* rejected whole, so everything in it got folded and the letter came out readable by accident. Judging each
* character on its own exposes the over-claim instead of masking it: `Thérèse` in Romanian read *ˈthrese*, the é
* gone, because the class promised a rule that did not exist. NATIVE_CLASS is a claim about the G2P, and
* `test/native-inventory.test.ts` now measures it rather than trusting it.
* ⚠ ä Ä ARE DELIBERATELY ABSENT: the g2p has no rule for them, and drops them outright —
* listing them here would promise a reading that does not exist. NATIVE_CLASS is a claim ABOUT
* THE G2P, and `test/native-inventory.test.ts` measures it character by character rather than
* trusting it.
*/
const NATIVE_CLASS = "[a-zãâîñçA-ZÃÂÎÑÇ]";
const nat = makeNativiser(NATIVE_CLASS, "u");
Expand Down
10 changes: 4 additions & 6 deletions src/languages/asturian/asturian.ts
Original file line number Diff line number Diff line change
Expand Up @@ -85,12 +85,10 @@ const TOKEN = new RegExp(`(${hostWordRun(["Latin"], "'·")})|(\\d+)|([.!?…,;:]
* language does not use, i.e. a foreign name. See core/hostWord.ts: this is the INVENTORY question, and it
* is no longer also deciding where the script boundary falls.
*
* ⚠ NOT QUITE VERBATIM: ḷ Ḷ were REMOVED, because the g2p has no rule for them and DROPPED them outright.
* The old token class listed them anyway, and the word-level fold hid the mismatch — a word containing one was
* rejected whole, so everything in it got folded and the letter came out readable by accident. Judging each
* character on its own exposes the over-claim instead of masking it: `Thérèse` in Romanian read *ˈthrese*, the é
* gone, because the class promised a rule that did not exist. NATIVE_CLASS is a claim about the G2P, and
* `test/native-inventory.test.ts` measures it rather than trusting it.
* ⚠ ḷ Ḷ ARE DELIBERATELY ABSENT: the g2p has no rule for them, and drops them outright —
* listing them here would promise a reading that does not exist. NATIVE_CLASS is a claim ABOUT
* THE G2P, and `test/native-inventory.test.ts` measures it character by character rather than
* trusting it.
*/
const NATIVE_CLASS = "[a-zñáéíóúüïḥA-ZÑÁÉÍÓÚÜÏḤ'·]";
const nat = makeNativiser(NATIVE_CLASS, "u");
Expand Down
9 changes: 4 additions & 5 deletions src/languages/catalan/normalize.ts
Original file line number Diff line number Diff line change
Expand Up @@ -222,11 +222,10 @@ export function normalizeCatalan(input: string): string {
// letter name (`B&Bs` → be i bes — the corpus's only ampersand is the plural). A TRUE minus (`-5`)
// reads "menys"; the corpus's `-\d` are all ranges/scores (6-6, 11.000-22.500, 4.2-3.9) and stay as
// two bare numbers.
// ⚠ ± IS THIS LANGUAGE'S OWN TWO WORDS, juxtaposed — zero new sourcing. Both halves are lifted from
// the plus and minus rules in this file, so nothing is invented, and both are SIGN names rather than
// operation names, which is what ± needs: it marks a tolerance, not an addition. The FORM is the one every
// language that already read ± uses (bg/da/is/nb/ro/sv juxtapose with no conjunction). Runs BEFORE the +
// rule, since ± is a single character the + rule cannot see.
// ⚠ ± IS A SINGLE CHARACTER (U+00B1), NOT A `+`, so no `+` rule can ever match inside it. It needs
// its own rule or the sign is dropped in silence; ordering against the `+` rule is free. The
// reading is this language's own two words juxtaposed, and ⚠ both are SIGN names rather than
// OPERATION names, which is what ± needs: it marks a TOLERANCE, not an addition.
s = s.replace(/±/gu, " més menys ");
s = s.replace(/\+\s?(?=\d)/gu, " més ");
s = s.replace(/(?<![\p{L}\p{Nd}])-(\d+)(?!\s*[-\d])/gu, "menys $1");
Expand Down
9 changes: 4 additions & 5 deletions src/languages/croatian/normalize.ts
Original file line number Diff line number Diff line change
Expand Up @@ -281,11 +281,10 @@ export function normalizeCroatian(input: string): string {
s = s.replace(/(?<!\p{L}\p{M})(\p{Lu})&(\p{Lu})(?![^\p{L}\p{M}])/gu, "$1 i $2");
s = s.replace(/\s&\s/gu, " i ");
s = s.replace(/(?<=\d)\s?[x×]\s?(?=\d)/gu, " puta ");
// ⚠ ± IS THIS LANGUAGE'S OWN TWO WORDS, juxtaposed — zero new sourcing. Both halves are lifted from
// the plus and minus rules already in this file, so nothing is invented. The FORM is the one every
// language that already read ± uses (bg/da/is/nb/ro/sv all juxtapose with no conjunction; English is the
// outlier that needs "or", and it already has its own rule). Runs BEFORE the + rule: ± is a single
// character, so the + rule cannot see it, and putting it first keeps the sign audible either way.
// ⚠ ± IS A SINGLE CHARACTER (U+00B1), NOT A `+`, so no `+` rule can ever match inside it. It needs
// its own rule or the sign is dropped in silence; ordering against the `+` rule is free. The
// reading is this language's own two words juxtaposed, both taken from the plus and minus rules
// already in this file.
s = s.replace(/±/gu, " plus minus ");
s = s.replace(/(^|[\s(])\+\s?(\d)/gu, "$1plus $2");
s = s.replace(/(?<=[A-Z])\+(\d)/gu, " plus $1");
Expand Down
9 changes: 4 additions & 5 deletions src/languages/czech/normalize.ts
Original file line number Diff line number Diff line change
Expand Up @@ -336,11 +336,10 @@ export function normalizeCzech(input: string): string {
s = s.replace(/(\d+)\s?°\s?C(?![\p{L}\p{M}])/gu, (_m, n: string) => `${n} ${counted(Number(n), DEGREE)} Celsia`);
s = s.replace(/(\d+)\s?°\s?F(?![\p{L}\p{M}])/gu, (_m, n: string) => `${n} ${counted(Number(n), DEGREE)} Fahrenheita`);
s = s.replace(/(\d+)\s?°/gu, (_m, n: string) => `${n} ${counted(Number(n), DEGREE)}`);
// ⚠ ± IS THIS LANGUAGE'S OWN TWO WORDS, juxtaposed — zero new sourcing. Both halves are lifted from
// the plus and minus rules already in this file, so nothing is invented. The FORM is the one every
// language that already read ± uses (bg/da/is/nb/ro/sv all juxtapose with no conjunction; English is the
// outlier that needs "or", and it already has its own rule). Runs BEFORE the + rule: ± is a single
// character, so the + rule cannot see it, and putting it first keeps the sign audible either way.
// ⚠ ± IS A SINGLE CHARACTER (U+00B1), NOT A `+`, so no `+` rule can ever match inside it. It needs
// its own rule or the sign is dropped in silence; ordering against the `+` rule is free. The
// reading is this language's own two words juxtaposed, both taken from the plus and minus rules
// already in this file.
s = s.replace(/±/gu, " plus mínus ");
s = s.replace(/(^|[\s(])\+\s?(?=\d)/gu, "$1plus ");
s = s.replace(/(\d)\s*×\s*(?=\d)/gu, "$1 krát ");
Expand Down
6 changes: 3 additions & 3 deletions src/languages/dutch/normalize.ts
Original file line number Diff line number Diff line change
Expand Up @@ -222,9 +222,9 @@ export function normalizeDutch(input: string): string {

// 9b) ± AND THE RELATIONAL AND DIVISION SIGNS. ± is this language's own two words juxtaposed, and
// the `plus` half is lifted from the rule directly above; `min` is not otherwise in this file precisely
// because nl_nl has no true minus sign (see the note above), so `min` is the one word here that comes
// from the register rather than from the tree. It is the standard Dutch reading of the sign and matches
// the juxtaposed form every language that already read ± uses (bg/da/is/nb/ro/sv).
// because this language has no true minus sign (see the note above), so `min` is the one word here
// that comes from the REGISTER rather than from another rule in this file. It is the standard Dutch
// reading of the sign.
s = s.replace(/±/gu, " plus min ");

// ⚠ THE ONLY LANGUAGE IN THIS BATCH WHERE TIER 2 SETTLED THE READING ON ITS OWN. Counted in nl_nl:
Expand Down
9 changes: 4 additions & 5 deletions src/languages/french/normalize.ts
Original file line number Diff line number Diff line change
Expand Up @@ -207,11 +207,10 @@ export function normalizeFrench(input: string, isWord: (lower: string) => boolea
// Emits the ordinary spelling `plus`; the HETERONYM map in french.jsonc supplies the [plys]
// operator reading, selected by the number that follows. This replaced a "plusse" respelling that
// existed only because Lexique carries just the [ply] "more" reading.
// ⚠ ± IS THIS LANGUAGE'S OWN TWO WORDS, juxtaposed — zero new sourcing. Both halves are lifted from
// the plus and minus rules already in this file, so nothing is invented. The FORM is the one every
// language that already read ± uses (bg/da/is/nb/ro/sv all juxtapose with no conjunction; English is the
// outlier that needs "or", and it already has its own rule). Runs BEFORE the + rule: ± is a single
// character, so the + rule cannot see it, and putting it first keeps the sign audible either way.
// ⚠ ± IS A SINGLE CHARACTER (U+00B1), NOT A `+`, so no `+` rule can ever match inside it. It needs
// its own rule or the sign is dropped in silence; ordering against the `+` rule is free. The
// reading is this language's own two words juxtaposed, both taken from the plus and minus rules
// already in this file.
s = s.replace(/±/gu, " plus moins ");
s = s.replace(/(\S)\+\s?(\d)/gu, "$1 plus $2");
s = s.replace(/(^|\s)\+\s?(\d)/gu, "$1plus $2");
Expand Down
9 changes: 4 additions & 5 deletions src/languages/german/normalize.ts
Original file line number Diff line number Diff line change
Expand Up @@ -148,11 +148,10 @@ export function normalizeGerman(input: string): string {

// 6) SIGNS.
s = s.replace(/(^|[\s(])[-−–](\d)/gu, "$1minus $2");
// ⚠ ± IS THIS LANGUAGE'S OWN TWO WORDS, juxtaposed — zero new sourcing. Both halves are lifted from
// the plus and minus rules already in this file, so nothing is invented. The FORM is the one every
// language that already read ± uses (bg/da/is/nb/ro/sv all juxtapose with no conjunction; English is the
// outlier that needs "or", and it already has its own rule). Runs BEFORE the + rule: ± is a single
// character, so the + rule cannot see it, and putting it first keeps the sign audible either way.
// ⚠ ± IS A SINGLE CHARACTER (U+00B1), NOT A `+`, so no `+` rule can ever match inside it. It needs
// its own rule or the sign is dropped in silence; ordering against the `+` rule is free. The
// reading is this language's own two words juxtaposed, both taken from the plus and minus rules
// already in this file.
s = s.replace(/±/gu, " plus minus ");
s = s.replace(/(\S)\+\s?(\d)/gu, "$1 plus $2");
s = s.replace(/(^|\s)\+\s?(\d)/gu, "$1plus $2");
Expand Down
9 changes: 4 additions & 5 deletions src/languages/greek/normalize.ts
Original file line number Diff line number Diff line change
Expand Up @@ -394,11 +394,10 @@ export function normalizeGreek(input: string): string {

// 11) SIGNS and VULGAR FRACTIONS. `(UTC +1)`; and `29¾ επί 24½ ίντσες`, where the elided noun is
// feminine (ίντσα) — «είκοσι εννιά και τρία τέταρτα».
// ⚠ ± IS THIS LANGUAGE'S OWN TWO WORDS, juxtaposed — zero new sourcing. Both halves are lifted from
// the plus and minus rules already in this file, so nothing is invented. The FORM is the one every
// language that already read ± uses (bg/da/is/nb/ro/sv all juxtapose with no conjunction; English is the
// outlier that needs "or", and it already has its own rule). Runs BEFORE the + rule: ± is a single
// character, so the + rule cannot see it, and putting it first keeps the sign audible either way.
// ⚠ ± IS A SINGLE CHARACTER (U+00B1), NOT A `+`, so no `+` rule can ever match inside it. It needs
// its own rule or the sign is dropped in silence; ordering against the `+` rule is free. The
// reading is this language's own two words juxtaposed, both taken from the plus and minus rules
// already in this file.
s = s.replace(/±/gu, " συν μείον ");
s = s.replace(/(?<![\p{L}\p{M}\d])\+\s?(?=\d)/gu, "συν ");
s = s.replace(/(\d)\s?½/gu, "$1 και μισή");
Expand Down
8 changes: 4 additions & 4 deletions src/languages/hungarian/normalize.ts
Original file line number Diff line number Diff line change
Expand Up @@ -464,10 +464,10 @@ export function normalizeHungarian(input: string): string {
s = s.replace(/(?<![\p{L}\p{M}\p{Nd}])[-−–](?=\d)/gu, (m0: string, off: number, whole: string) =>
/\d\s*$/u.test(whole.slice(0, off)) ? m0 : "mínusz ",
);
// ⚠ ± IS NOW FREE, and it was not before this commit: it needs two SIGN names and this file had only the
// plus until the minus rule above was added. Both halves are lifted from rules in this file, so nothing is
// invented, and the FORM is the juxtaposition every language that already reads ± uses. Runs BEFORE the
// + rule, since ± is a single character the + rule cannot see.
// ⚠ ± TAKES TWO SIGN NAMES, so it is only expressible once BOTH the plus and the minus rules exist —
// both halves are taken from the rules in this file. ⚠ It needs its OWN rule: ± is a single character
// (U+00B1), not a `+`, so no `+` rule can match inside it and the sign would otherwise be dropped in
// silence.
// ⚠ AND hu.wikipedia NAMES BOTH SIGNS TOGETHER, which is as direct as this gets: "A két előjel a
// pluszjel (+) és a mínuszjel (−), melyek a matematikában a pozitív és a negatív fogalmát" — the two
// SIGNS are the plus sign and the minus sign, expressing positive and negative. Exactly the sense ±
Expand Down
9 changes: 4 additions & 5 deletions src/languages/indonesian/normalize.ts
Original file line number Diff line number Diff line change
Expand Up @@ -100,11 +100,10 @@ export function normalizeIndonesian(input: string): string {

// 6) SIGNS. Neither occurs in this corpus, but a dropped sign is silent content loss wherever it does.
s = s.replace(/(^|[\s(])[-−–](\d)/gu, "$1minus $2");
// ⚠ ± IS THIS LANGUAGE'S OWN TWO WORDS, juxtaposed — zero new sourcing. Both halves are lifted from
// the plus and minus rules already in this file, so nothing is invented. The FORM is the one every
// language that already read ± uses (bg/da/is/nb/ro/sv all juxtapose with no conjunction; English is the
// outlier that needs "or", and it already has its own rule). Runs BEFORE the + rule: ± is a single
// character, so the + rule cannot see it, and putting it first keeps the sign audible either way.
// ⚠ ± IS A SINGLE CHARACTER (U+00B1), NOT A `+`, so no `+` rule can ever match inside it. It needs
// its own rule or the sign is dropped in silence; ordering against the `+` rule is free. The
// reading is this language's own two words juxtaposed, both taken from the plus and minus rules
// already in this file.
s = s.replace(/±/gu, " plus minus ");
s = s.replace(/(\S)\+\s?(\d)/gu, "$1 plus $2");
s = s.replace(/(^|\s)\+\s?(\d)/gu, "$1plus $2");
Expand Down
9 changes: 4 additions & 5 deletions src/languages/italian/normalize.ts
Original file line number Diff line number Diff line change
Expand Up @@ -213,11 +213,10 @@ export function normalizeItalian(input: string): string {
// 8) SIGNS. `+` occurs once (`UTC+1`); `-` does not occur in this corpus, but a dropped minus is silent
// content loss that inverts a temperature, and the guards keep it off the ranges that DO occur —
// `1894-1895` has a digit before the hyphen, and the football score `26 - 00` has a space after it.
// ⚠ ± IS THIS LANGUAGE'S OWN TWO WORDS, juxtaposed — zero new sourcing. Both halves are lifted from
// the plus and minus rules immediately below, so nothing is invented. The FORM is the one every language
// that already read ± uses (bg/da/is/nb/ro/sv all juxtapose with no conjunction; English is the outlier
// that needs "or", and it already has its own rule). Runs BEFORE the + rule: ± is a single character, so
// the + rule cannot see it, and putting it first keeps the sign audible either way.
// ⚠ ± IS A SINGLE CHARACTER (U+00B1), NOT A `+`, so no `+` rule can ever match inside it. It needs
// its own rule or the sign is dropped in silence; ordering against the `+` rule is free. The
// reading is this language's own two words juxtaposed, both taken from the plus and minus rules
// immediately below.
s = s.replace(/±/gu, " più meno ");
s = s.replace(/(\S)\+\s?(\d)/gu, "$1 più $2");
s = s.replace(/(^|\s)\+\s?(\d)/gu, "$1più $2");
Expand Down
9 changes: 4 additions & 5 deletions src/languages/japanese/normalize.ts
Original file line number Diff line number Diff line change
Expand Up @@ -210,11 +210,10 @@ export function normalizeJapanese(input: string): string {

// 9) SIGNS. Neither occurs in this corpus, but a dropped sign is silent content loss wherever it does.
s = s.replace(/(^|[\s((])[-−–](\d)/gu, "$1マイナス$2");
// ⚠ ± IS THIS LANGUAGE'S OWN TWO WORDS, juxtaposed — zero new sourcing. Both halves are lifted from
// the plus and minus rules already in this file, so nothing is invented. The FORM is the one every
// language that already read ± uses (bg/da/is/nb/ro/sv all juxtapose with no conjunction; English is the
// outlier that needs "or", and it already has its own rule). Runs BEFORE the + rule: ± is a single
// character, so the + rule cannot see it, and putting it first keeps the sign audible either way.
// ⚠ ± IS A SINGLE CHARACTER (U+00B1), NOT A `+`, so no `+` rule can ever match inside it. It needs
// its own rule or the sign is dropped in silence; ordering against the `+` rule is free. The
// reading is this language's own two words juxtaposed, both taken from the plus and minus rules
// already in this file.
s = s.replace(/±/gu, " プラスマイナス ");
s = s.replace(/(^|[\s((])\+\s?(\d)/gu, "$1プラス$2");
s = s.replace(/(\S)\+\s?(\d)/gu, "$1プラス$2");
Expand Down
Loading
Loading