Should search collation be a different data key + constructor?

[sffc]

It seems that the decision for whether to use search collations is when the Collator is created in order to perform equality operations on strings. This is the type of thing that the developer could specify in code:

let ordering_collator = Collator::try_new_unstable(...);
let equality_collator = Collator::try_new_search_unstable(...);

Under the hood, we could keep all the same code paths, but move the search collations into a different data key so that static data slicing can pick it up. This would help simplify datagen because we wouldn't need the search* stuff; we just look for the collations/search@1 data key.

CC @hsivonen

[hsivonen]

If we introduce a separate constructor, I think it should also construct a distinct type that has matches that returns bool instead of having compare that returns Ordering. As an implementation matter, this distinct type should just wrap a Collator without introducing extra fields.

Unfortunately, for JavaScript ECMA-402 compat, there should still be some way to do the wrong thing and extract the wrapped Collator. (The alternative would be that ICU4X-backed Intl.Collator with usage: "search" would only return 0 and 1 and never -1, or 0 and -1 and never 1. While this would be correct in principle, it would break potential pre-existing misuse of usage: "search" on the Web. Notably, mapbox-gl-js supports >, <, <= and >= predicates with a collator instantiated with usage: "search". Unclear if Web apps using mapbox-gl-js make use of that.)

[hsivonen]

Also, how to fit searchjl into this needs more thought than in ECMA-402. AFAICT, if you want searchjl, which is semantically search, you need to use usage: "sort", collation: "searchjl" when instantiating Intl.Collator.

[sffc]

I think including the search-specific key would imply search, and searchjl can be enabled via a datagen flag the same way as big5han.

I'm fine with an API on the SearchCollator to convert it to a regular Collator. I don't think we should get in the business of returning only -1/0 or 0/1 in ECMA-402; that's too risky.

What should be the name of SearchCollator?

• SearchCollator
• EqualityCollator
• Comparator
• EqualityComparator
• StringMatcher
• FuzzyStringMatcher

[hsivonen]

Of these, FuzzyStringMatcher is probably the closest to capturing the use case and SearchCollator is probably the closest to capturing the mechanism. Let's say FuzzyStringMatcher.

[sffc]

Discuss with:

• @sffc
• @echeran

Optional:

• @markusicu
• @hsivonen
• @Manishearth (curious and would like to listen in if possible)

[sffc]

• @hsivonen - We should think about the use case; right now we only have whole-string matching. The idea is you have a set of strings and a filter key and you filter down the set to the strings that match your filter key. How relevant that is for anything in reality is questionable. We looked for usages of ECMA-402 Search collation on the web, and almost all of them are one call site in mapbox. It's a framework that exposes a feature and it's not clear how web pages use it. When do you have a set of human-readable labels and filter down the string on a full match? I think it's not worthy of API. But I think it could make sense as a different key... so that we can support ECMA-402. So I see it more as a data optimization thing and not a genuine use case. A substring or prefix matching API could have use cases. For example, filtering an address book on a prefix match of a name token starts being a real use case. So that might be a better direction, even though it is not in ECMA-402.
• @markusicu - I agree that the use case without a string search API is questionable. If someone wanted to create a specific API for this sort of thing, I would give that either a specific name that says something about collation. If it just wants to be a trait, something like FuzzyStringMatcher could make sense, but it could be implemented by things that are not based on collation. I would not tie FuzzyStringMatcher to something that is always collator internally.
• @sffc - I want to determine what the use case is supposed to be.
• @markusicu - You can basically construct a searching and matching function on top of collation. We have specific data that is designed to improve that use case. For a lot of the characters, the vanilla collation data works well, but not for others. When you are searching, you care about equality, not the order. For example, the search tailoring for German does umlaut expansion. We have only one search tailoring even though we have two for order. See here in the spec: https://www.unicode.org/reports/tr10/#Searching. It's typically used for a substring match, like Ctrl-F in a browser.
• @hsivonen - Why is collation the way it is? There's a use case for diacritic-insensitive string matching. And there is also the observation that you need special handling for certain diacritics like German umlauts. This data is already in collator, so you can use it to build your search algorithm. It seems weird that Thai for example has certain tailorings that are not in other Brahmic languages. The ECMA-402 API really feels like, we have this data, so we may as well export it. But, if we had a prefix matching API, it would help with ECMA-402 and it would be a legitimate API of ICU4X. It's easy to implement, and much easier than a full string search.
• @sffc - What would be the building block API for a full string search?
• @hsivonen - It wouldn't be very efficient to use prefix match for that
• @markusicu - Also it could find matches in contractions and things. So prefix match is a useful API but it's not a building block for string search. But some of the things you need to think about for substring match you also need to think for prefix match.
• @hsivonen - It sounds like I haven't thought through the prefix match case as much as Markus.
• @sffc - Datagen already handles this filtering, but if we wanted to implement prefix/substring search, we might want to move this data to a new key.
• @hsivonen - If the existing datagen solution is satisfactory, I think we should leave things as they are.

Consensus: Leave things as they are; we are open to exploring a prefix or substring match API in the future when we have a clear client need.

LGTM: @markusicu @sffc @younies @echeran @hsivonen @robertbastian