Inconsistent locale options for constructors

[mihnita]

I would expect the code below to work:

use icu::locale::locale;

use icu::datetime::fieldsets::YMD;
use icu::datetime::DateTimeFormatter;

use icu::segmenter::WordSegmenter;

fn main() {
    let _formatter = DateTimeFormatter::try_new(
        locale!("en").into(),
        YMD::medium()
    );

    let _segmenter = WordSegmenter::try_new_auto(
        locale!("en").into()
    );
}

But it fails with

error[E0277]: the trait bound `WordBreakOptions<'_>: From<Locale>` is not satisfied
  --> src\main.rs:15:9
   |
15 |         locale!("en").into()
   |         ^^^^^^^^^^^^^ ---- required by a bound introduced by this call
   |         |
   |         the trait `From<Locale>` is not implemented for `WordBreakOptions<'_>`
   |
   = note: required for `Locale` to implement `Into<WordBreakOptions<'_>>`

[mihnita]

Following the example works:

use icu::locale::langid;
use icu::segmenter::options::WordBreakOptions;
...

  let mut options = WordBreakOptions::default();
  let langid = &langid!("en");
  options.content_locale = Some(langid);
  let segmenter = WordSegmenter::try_new_auto(options).unwrap();

But why the (gratuitous?) inconsistency?

---

WordSegmenter

https://docs.rs/icu/2.1.1/icu/segmenter/struct.WordSegmenter.html#method.try_new_auto

pub fn try_new_auto(
    options: WordBreakOptions<'_>,
) -> Result<WordSegmenter, DataError>

pub struct WordBreakOptions<'a> {
    pub content_locale: Option<&'a LanguageIdentifier>,
    pub invariant_options: WordBreakInvariantOptions,
}

DateTimeFormatter

https://docs.rs/icu/2.1.1/icu/datetime/struct.DateTimeFormatter.html#method.try_new

pub fn try_new(
    prefs: DateTimeFormatterPreferences,
    field_set_with_options: FSet,
) -> Result<DateTimeFormatter<FSet>, DateTimeFormatterLoadError>

pub struct DateTimeFormatterPreferences {
    pub locale_preferences: LocalePreferences,
    pub numbering_system: Option<NumberingSystem>,
    pub hour_cycle: Option<HourCycle>,
    pub calendar_algorithm: Option<CalendarAlgorithm>,
}

pub struct LocalePreferences { /* private fields */ }

---

What I would expect:

• both DateTimeFormatter & WordSegmenter take some kind of *Options or *Preferences, not the current mixed names (DateTimeFormatterPreferences / WordBreakOptions)
• the options bad should name the locale part the same: locale_preferences or content_locale
• and the locale part should take the same type langid / locale / LocalePreferences / LanguageIdentifier

Because the ideal would be that once you know how to do one i18n operation, you know them all (more or less).

[robertbastian]

"options" and "preferences" are inherently different:

• preferences are derived from user preferences, which is why they all implement From<Locale>
• options are the developer-chosen options for an API. As the identifier content_locale suggests, that option is the (optional) language of the content, not the locale of the user

Some APIs accept both options and preferences: https://docs.rs/icu/latest/icu/collator/struct.Collator.html#method.try_new

[mihnita]

"options" and "preferences" are inherently different:

The names don't communicate anything about that, they are just generic.

that option is the (optional) language of the content

That's completely inconsistent with everything else.

And it is not really the language of the content, it is the language of the segmenter.
I can use French segmenter on Japanese text. I will get crap results, but it is still the language of the segmenter.
That is why it is the passed to the constructor of the WordSegmenter.
If it was the language of the content it would be passed to the segment* methods, together with the content.

Last, even if you want to look at it as the language of the content, it is clunky to use different types (Locale vs LanguageIdentifier).
In ICU4C / 4J you can just use Locale (and in Java ULocale) everywhere.

I understand why ICU4X created 2 different types.
But I should be able to use them everywhere (the way in ICU4J I can use ULocale / Locale (almost) everywhere). In ICU Locale is from JDK, ULocale is ICU4J type, so the two are "from outside", ICU itself is consistent (in this respect).

---

There might be technical / implementations justifications for all of this.
But as a user it does not matter.
It makes the APIs unfriendly and not easily discoverable.
Consistency is good.

[sffc]

This design is the result of an extensive discussion about what "locale" means for text-oriented components like segmenter. You can read the discussion here: #3284

I just recently merged some improved docs: #7136

The observed error is doing its job of informing the client that the user locale should not be used when configuring a segmenter. The locale should instead be a hint derived from the text content.

[sffc]

@mihnita, can you verify if the improved docs in #7136 clarify this for you, in case you hadn't seen them yet (they weren't included in the 2.1 release)?

[mihnita]

can you verify if the improved docs in #7136 clarify this for you

No, it does not, not much.

It is good as documentation, but I should not have to read it.

In my opinion the ideal collection of APIs allow me to move between various functional areas without carefully reading pages and pages of documentation.
Once I know how to create a date formatter, and a collator, I also know how to create a number formatter, list formatter, word segementer, etc.

And the difference in APIs might make sense for implementation, but it is gratuitous for a user.

Ultimately in the current design WordSegmenter takes a LanguageIdentifier, and DateFormatter a Locale.
And Locale = LanguageIdentifier + extensions.

But LDML defines extensions that affect segmentation:

key	key description
dx	Dictionary break script exclusions
lb	Line break style
lw	Line break word handling
ss	Sentence break suppressions

https://www.unicode.org/reports/tr35/#Key_And_Type_Definitions_

It is very-very convenient to pass a locale to segmenter and "magically" have all preferences honored

Let's look at LineBreakOptions

pub struct LineBreakOptions<'a> {
    pub strictness: Option<LineBreakStrictness>,
    pub word_option: Option<LineBreakWordOption>,
    pub content_locale: Option<&'a LanguageIdentifier>,
}

If I get a locale (that's usually what you get, think a request to a server) and I want to do line breaking I have to take that locale, split the LanguageIdentifier part of it to use in content_locale, and inspect the extensions to set the strictness and word_option.

That is clunky and is not forward compatible.

If (for example) LDML adds another extension that affects segmentation, and I start getting that in requests, my server using icu4x can't "magically" honor it.
Because I didn't know I needed to inspect that extension, and map it to some kind of LineBreakWordOption.
I have to update my code.

If there would be a way to create a LineBreakOptions from a Locale (for example) that would work.

---

But as it is the APIs are inconsistent, and seem to expose decisions taken for implementation reasons.

Why not allow segmenters to take a Locale, and the implementation can ignore the extension part, for example?

You can read the discussion here: #3284

I did. And I see that both Mark and Markus made the same argument for Locale in segmenters that I did.
And they've been overridden without much explanation other than "we can always add it later".

"Add it later" is an easy way out for a library that can't make decisions looking at the big picture.

It comes at a cost for the developers: they must change their code to get correct functionality (see above about "forward compatible").

---

Nitpick: in segmenter(s) the field is named content_locale, but it is not a Locale, it is a LanguageIdentifier.

[sffc]

Thank you for the comments. Much appreciated. My reply below attempts to provide background on how ICU4X arrived at the current design, without taking a position on whether the design is "right" or "wrong".

---

You can read the discussion here: #3284

I did. And I see that both Mark and Markus made the same argument for Locale in segmenters that I did. And they've been overridden without much explanation other than "we can always add it later".

As chair of the TC, I take great pride in our open decision-making process by consensus. I reviewed the thread, and I don't see an example of Mark or Markus being "overridden without much explanation". There were 10 people who contributed to the discussion, from multiple Unicode groups. I encourage you to share feedback privately on how this decision-making process can be improved.

But LDML defines extensions that affect segmentation:

This is true. This is a point that was not raised in the thread.

My attempt at an explanation, which may or may not be satisfactory: ICU4X and ECMA-402 have both decided to support a subset of the locale extension keywords. They do not support those that are more about developer preference over user preference. The two line break options ICU4X supports were determined to be more about the developer preference, and therefore associating them with the user locale would have been the wrong design.

As for why content_locale is a LanguageIdentifier: because it identifies the language of the content, nothing more. It is not a user locale with user preferences. It simply tags text as being written in a particular language.

Nitpick: in segmenter(s) the field is named content_locale, but it is not a Locale, it is a LanguageIdentifier.

Sure, maybe content_language would have been a better name.

---

My personal opinion: I've seen enough evidence in ECMA-402 that the Intl.Segmenter locale option is often used the wrong way, passing a user locale instead of a text content locale. (Intl.Collator has the same problem.) I believe that content_locale helps solve it.

[mihnita]

I believe that content_locale helps solve it.

I strongly doubt that. It is just a name...

Even that's weird. content_locale has "locale" in name. But takes a langid, not a locale.

And it really depends how you look at it.

When I create a segmenter("th") and I try to segment Japanese I will get crap results.
The "th" there is the locale of the segmenter, not of the content.

When I created it it loaded Thai segmentation rules. It IS a Thai segmenter.
The same way I spell check French content with a French dictionary.
If the locale of the dictionary and the locale of the content match, I get good results.
And I would argue that this is how most people think.
Changing that pattern might help some, and confuse others.

If this is the locale of the content then that would be passed with the content:

let segmenter = WordSegmenter::new();
let seg = segmenter.segment(text, locale);

---

The thing is, if decisions are taken piece-meal, one at the time, no matter how reasonable in isolation, the resulting API is a hot mess.

Let's take a look:

** CaseMapper **

let cm = CaseMapper::new();
cm.uppercase_to_string("hello world", &langid!("en"))

There is a langid (not a locale), and it is passed with the content, not with the CaseMapper

---

Collator

pub fn try_new(
    prefs: CollatorPreferences,
    options: CollatorOptions,
) -> Result<CollatorBorrowed<'static>, DataError>

So it takes Preferences and Options. These terms are so close that the separation is basically meaningless. I choose an option or another based on my preferences :-)

But let's go on:

pub struct CollatorPreferences {
    pub locale_preferences: LocalePreferences,
    pub collation_type: Option<CollationType>,
    pub case_first: Option<CollationCaseFirst>,
    pub numeric_ordering: Option<CollationNumericOrdering>,
}

So here we have a locale_preferences, not content_locale.
Why?
I'm sorting content, and this is the locale of that content.
Same argument you offered for Segmenters. But does not apply to Collator, for some reason.

---

ListFormatter

pub fn try_new_and(
    prefs: ListFormatterPreferences,
    options: ListFormatterOptions,
) -> Result<ListFormatter, DataError>

pub fn try_new_and(
    prefs: ListFormatterPreferences,
    options: ListFormatterOptions,
) -> Result<ListFormatter, DataError>

pub struct ListFormatterPreferences {
    pub locale_preferences: LocalePreferences,
}

So this is consistent with Collator.

---

PluralRules

pub fn try_new(
    prefs: PluralRulesPreferences,
    options: PluralRulesOptions,
) -> Result<PluralRules, DataError>

pub struct PluralRulesPreferences {
    pub locale_preferences: LocalePreferences,
}

So this is consistent with Collator and ListFormatter.

---

**DecimalFormatter**

```rust
pub fn try_new(
    prefs: DecimalFormatterPreferences,
    options: DecimalFormatterOptions,
) -> Result<DecimalFormatter, DataError>


pub struct DecimalFormatterPreferences {
    pub locale_preferences: LocalePreferences,
    pub numbering_system: Option<NumberingSystem>,
}

So this is consistent with Collator, ListFormatter, and PluralRules.

pub fn try_new_auto(
    options: WordBreakOptions<'_>,
) -> Result<WordSegmenter, DataError>

pub struct WordBreakOptions<'a> {
    pub content_locale: Option<&'a LanguageIdentifier>,
    pub invariant_options: WordBreakInvariantOptions,
}

So this one has no *Preferences, only *Options*. The locale part is in Options, not in Preferences, like in all the other APIs. The locale setting is called content_locale, not locale_preferences, inconsistent with the other APIs. And despite the name (content_locale) the type is LanguageIdentifier. That is internal consistency. I would expect that the type of something called foo_locale is a Locale`.

What's the difference between segmenters and collator, to justify this big differences?

---

Like everything else in software (and not only) there is no perfect solution, it is all a matter of compromises.
Speed, size, modularity, ease of use, consistency, etc.

Consistency also helps with ease of use.

The documentation for icu4x is really good, with lots of examples.

But I don't want to read it unless I don't understand how to use something.
And "how we decided this" is something that I never want to read, as a user.
I might read some "why do it this way" if I am really interested in i18n and in that library.

Ideally, if I am already familiar with a library, I should understand how to use a new piece of that library without reading docs of history.

[mihnita]

You can read the discussion here: #3284

I reviewed (again) that thread.

Everybody said "locale", in all comments.
I don't see anything there about langid vs locale, content_locale vs locale_preferences, Options vs Preferences, etc.

So in the end that discussion didn't help clarify the reasons for the API as it is.

[sffc]

"Preferences" is a user locale with structured fields, introduced in ICU4X 2.0. "Options" are things that should be set by the developer based on application requirements, not user preferences. This naming is used consistently across all components.

The content locale is in the constructor of Segmenter because it impacts data loading, and in the terminal function of CaseMapper because it does not impact data loading. This did result in an unfortunate inconsistency. #3234 (comment)

In Collator, the user locale matters sometimes; for example, you might have a contact list with names from various different languages, but you sort them according to your language. #6033

I put this issue into the 3.0 milestone when we can re-evaluate these questions.

[robertbastian]

Segmenter and case mapper are two exceptions where we use content languages instead of user locales, because it makes sense in those contexts (they should not be sensitive to user preferences). Yes the field should have probably been called content_language. Minor.

the resulting API is a hot mess

We've spent a lot of time designing this, and the API we have come up with is principled and satisfies our constraints. You were not part of any of those discussions, so you don't get to say things like this.