fix(analyzer): locale-aware case fold for Turkish context words - #2208
Open
fevziegeyurtsevenler wants to merge 1 commit into
Open
fix(analyzer): locale-aware case fold for Turkish context words#2208fevziegeyurtsevenler wants to merge 1 commit into
fevziegeyurtsevenler wants to merge 1 commit into
Conversation
Signed-off-by: Fevzi Ege Yurtsevenler <egeyurtsevenler@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change Description
LemmaContextAwareEnhancerlowercases context words and surrounding lemmas withstr.lower()before comparing them.str.lower()is locale-independent, and forTurkish/Azerbaijani it produces the wrong result for the dotted/dotless I:
Because of this, a context word written in uppercase (which is how Turkish ID and
vehicle documents are usually printed) no longer contains the recognizer's context
term, so the context confidence boost is never applied for that input.
This is visible with the bundled Turkish recognizers, whose context lists are
lowercase Turkish (
"tc kimlik","kimlik no","kayıt", …): those words nevermatch their own uppercase form. (These recognizers are country-specific and ship
enabled: falseindefault_recognizers.yaml, so this only affects users who optinto them — the point is that once enabled, their context matching is locale-broken
for uppercase input.)
Fix
Add a small
_fold(text, language)helper on the enhancer. When the recognizer'ssupported_languageistr/az, the dotted/dotless I pairs are pre-mapped(
İ→i,I→ı) before lowering; for every other language (and when no language isgiven)
_foldreturns exactlytext.lower(). The language is taken from thematched recognizer.
Which sites fold, which stay plain. The old up-front
context = [word.lower() for word in context]at the top ofenhance_using_contextis removed — folding the caller-supplied
contextearly would discard theoriginal casing that the locale-aware fold needs, so those words are now folded at
comparison time instead. The two comparison sites in
_find_supportive_word_in_context(thesubstringbranch and thewhole_wordbranch) fold both operands through
_fold, and the stored surrounding wordappended in
_add_n_wordsis folded as well. One.lower()deliberately staysplain: the membership test
lemmas[i].lower() in lemmatized_filtered_keywordsin_add_n_words, because those keywords are the plain-str.lower()keywords built byNlpArtifactsand must be compared like-for-like — folding only the collected wordthat is later compared against the recognizer context.
Zero blast radius for non-tr/az: for any other language
_fold(x, lang)isbyte-for-byte identical to
x.lower(), so existing behaviour is unchanged.Why this lives in the core enhancer (and why
casefold()is not enough)The fold is applied once, in
LemmaContextAwareEnhancer, rather than in eachTurkish recognizer. The lowering that breaks the match happens inside the
enhancer — recognizers only declare their lowercase context lists and never see the
surrounding-word lowering, so a recognizer has no seam at which to intervene.
Putting the rule in each recognizer would duplicate it across every current and
future
tr/azrecognizer and still could not touch the enhancer's ownstr.lower()on surrounding lemmas. Keying off the recognizer's already-declaredsupported_languagekeeps the rule in exactly one place and leaves every otherlanguage on the identical code path.
Switching the existing calls to
str.casefold()does not fix this.casefold()is also locale-independent:
"TC KİMLİK NO".casefold()still yields'tc ki̇mli̇k no'(with the combining dot) and"KAYIT".casefold()still yields'kayit', not'kayıt'. The dotted/dotless I has to be mapped explicitly fortr/az, which is what_folddoes before lowering.Tests
Failing-then-fix unit tests were added for both bundled Turkish recognizers,
asserting that uppercase Turkish context raises the score above the pattern
baseline and records the matching context word:
test_tr_national_id_recognizer.py:"TC KİMLİK NO"supplied as context.test_tr_license_plate_recognizer.py:"KAYIT"as a surrounding lemma(exercising the lemma-extraction path).
Both assertions fail on
main(score stays at the baseline) and pass with the fix.A unit test for the
whole_wordbranch was also added totest_lemma_context_aware_enhancer.py(
test_when_whole_word_turkish_context_then_locale_aware_match) so the Turkishpath of the
whole_wordfold site is asserted directly:"KAYIT"matches"kayıt"withlanguage="tr"/"az", and does not matchwithout a locale. The remaining existing
test_lemma_context_aware_enhancer.pyandtest_context_support.pycasescontinue to pass unchanged.
Issue reference
No pre-existing issue on the tracker; the defect and a runnable reproduction are
described in the Change Description above.
Checklist