Add non-complex segmenter constructors

[robertbastian]

Users already do this with custom data (i.e. linebender/parley#436), we should provide an easier way.

Fixes #3612

[Manishearth]

A bit worried about the large number of ctors we have on Segmenter.

@sffc , thoughts?

[sffc]

There's already an issue for this: #3612

I don't like the name "empty". Maybe it's okay. Since the only thing this PR does is add the new constructors, I don't see value in merging it until we've agreed on the name for it.

[sffc]

Also, reviewing the Parley thread, I don't think they should be using these new constructors? One of the reasons they want to use icu_segmenter is because it supports non-rule-based segmentation. Disabling it seems counterproductive. If they are observing a multi-MB size increase, it means they might be linking the dictionary data or something. The LSTM is much less than that.

[robertbastian]

I don't like the name "empty"

The "empty" constructor is on the internal ComplexPayloads type, it's not public API.

One of the reasons they want to use icu_segmenter is because it supports non-rule-based segmentation. Disabling it seems counterproductive.

They do want to put complex segmentation behind a feature flag or BYOD though, which will be much easier if they don't have to do this through a single API by conditionally changing data provider behaviour.

[sffc]

This is good modulo the function name which we can discuss today.

naming resolved

[sffc]

Would you mind listing the new APIs added by this PR?

I think it is the new_for_non_complex_scripts constructors and some default fns on options structs that already implemented the Default trait. Right?

[robertbastian]

Changed APIs:

• new LineSegmenter::new_for_non_complex_scripts + buffer/unstable variants
• new WordSegmenter::new_for_non_complex_scripts + buffer/unstable variants
• LineBreakOptions::default becomes const
• WordBreakOptions::default becomes const
• WordBreakInvariantOptions::default becomes const

[mihnita]

And that's how a library ends up with inconsistent, untidy APIs.

One of the basic rules for an i18n library should be that all locales are treated the same when I look at the public APIs.
In the implementation a lib can look and say: this locale is for a non-complex script (or requires a dictionary, or has an ML model, whatever) and treat it differently.

As a dev I should not know what kind of a locale something is.
Otherwise I am forced to write code like this:

if is_complex_script(locale)
   seg = WordSegmenter.new_for_non_complex_scripts(locale)
else
   seg = WordSegmenter.new_some_other_kind(locale)

And how do I know if a locale "is_complex_script"?
I hard-code a list somewhere?

The proper solution is to hide all of this in the regular constructor, in the library:

new_universal_constructor(locale)
   if is_complex_script(locale)
      seg = WordSegmenter.new_for_non_complex_scripts(locale)
   else
      seg = WordSegmenter.new_some_other_kind(locale)

private new_for_non_complex_scripts(...)
private new_some_other_kind(...)

[robertbastian]

Otherwise I am forced to write code like this

The proper solution is to hide all of this in the regular constructor, in the library

You don't seem to have all the context here. We have the proper solution, our main constructor is called try_new_auto and automatically handles all scripts. This constructor is specifically for clients who want to not pay the binary and data size cost of including LSTMs or dictionaries because they know that they will never need to segment text in complex languages (which are listed in the docs). Those users exist, not every project is localized into all the world's languages or renders arbitrary user input.