`icu_casemapping` feature request: methods `fold` and `full_fold` should apply Turkic mappings depending on locale

[lucatrv]

I find awkward that I need to manually verify locale and apply either methods fold/full_fold or fold_turkic/full_fold_turkic, IMHO there should be only methods fold/full_fold and they should automatically apply Turkic (T) mappings depending on locale.

[Manishearth]

cc @echeran to look into and respond to this

[markusicu]

Applying Turkic case foldings automatically is dangerous.

While case mappings are intended for human consumption and take a locale parameter, case foldings are used for processing (case-insensitive matching) not for display, and in most cases it is very surprising when "IBM" and "ibm" don't match when the locale is Turkish or Azerbaijani.

It is much safer to let the developer control this explicitly. (By comparison, ICU4C/ICU4J have folding functions that take a boolean parameter for default vs. Turkic foldings. This also models the boolean condition in the relevant Unicode data file.)

[lucatrv]

@markusicu thanks for your clarification, I now understand that for case mappings I should use CaseMapping::to_full_lowercase instead of CaseMapping::to_full_fold, and why there is a separate full_fold_turkic method. Maybe a similar explanation in icu_casemapping documentation would be useful also for others.

[Manishearth]

Reopening for the documentation fix

[lucatrv]

Thanks to reopening this issue for documentation, anther point that IMHO would be useful to clarify is when to use a case-folding method of icu_casemapping and when to use icu_collator to compare / match strings.

If I understand correctly, icu_collator should be used when strings need to be sorted, while a case-folding method of icu_casemapping should be used when strings need just to be matched. However icu_collator can also be used to match strings, see for instance examples using Ordering::Equal here, so it is not clear to me which one to use in this case.

Finally, another source of confusion (at least for me) is that icu_casemapping can be used for both case mapping and case folding, but its documentation mentions only "Case mapping for Unicode characters and strings".

[sffc]

The collator does a fuzzier match. The example you cited shows that it considers "às" and "as" to be equal, for example.

@markusicu is it safe to say that most users who are looking for a fuzzy string comparison utility should favor the collator over casefold+nfd?

[sffc]

CC @hsivonen

Also see related discussion in 402 last month, tc39/ecma402#256 (comment)

[hsivonen]

Casefold+NFD and ignoring combining diacritics after the NFD operation gives a general case-insensitive, diacritic-insensitive match. To further match the root search collation (apart from the Hangul aspect for which I don't understand the use case), you'd have to also ignore certain Arabic marks and the Thai phinthu (virama). (The Hebrew aspect of the search root is gone from CLDR trunk already.)

Apart from Turkic case-insensitivity, the key thing that the search collation tailorings provide on top of the above is being able to have a diacritic-insensitive mode where certain things that technically are diacritics but that are on a per-language basis considered to form a distinct base letter are not ignored on a locale-sensitive basis. For example, o and ö are distinct for Finnish, Swedish, Icelandic, and Turkish (not sure if them being equal for Estonian search is intentional or a CLDR bug) in collator-based search even when ignoring diacritics.

Based on observing the performance of Firefox's ctrl/cmd-f (not collator based) relative to Chrome's and Safari's (collator-based), I believe that casefold+NFD and ignoring certain things post-NFD will be faster than collator-based search. However, if you also want not to ignore certain diacritics on a per-locale basis, it's up to you to implement those rules. That is, ICU4X doesn't do it for you. You can find out what the rules are by reading the CLDR search collation sources.

(FWIW, Firefox's ctrl/cmd-f does not have locale-dependent rules for diacritics. The checkbox either ignores all of them or none.)

ECMA-402 and ICU4X don't have API surface for collator-based substring match. You can only do full-string comparison, so you can search in the sense of filtering a set/list of items by a search key.

[hsivonen]

Thai phinthu (virama)

Correction: To match the search root, you'd also have to ignore everything that has zero primary weight in the sort root. That is, all the characters whose line in https://www.unicode.org/Public/UCA/15.0.0/allkeys.txt matches ; [.0000.

[lucatrv]

If I understand correctly, CaseMapping::to_full_fold applies full case folding + NFD and ignores combining diacritics. Therefore it actually provides the fuzziest match (general case-insensitive and diacritic-insensitive match).
To my understanding this should be equivalent to the icu_collator primary strength level, which I guess is independent from locale same as CaseMapping::to_full_fold.
What icu_collator provides on top of this are the five configurable strength levels and locale-aware ordering.

Is this more or less correct?...