State of Unicode in Rust

[robertbastian]

The point of this issue is to track the implementation of Unicode algorithms in Rust, specifically wrt compatibility with ICU4X.

The Unicode Technical Reports are listed at https://www.unicode.org/reports/. Some identify as Unicode Standard Annex (UAX) or Unicode Technical Standard (UTS).

.	Name	Status
UAX 9	Unicode Bidirectional Algorithm	✅ implemented in unicode-bidi, can use icu::properties data
UTS 10	Unicode Collation Algorithm	✅ implemented in icu::collator
UAX 11	East Asian Width	⚠️ implemented in unicode-width, cannot use icu::properties data
UAX 14	Unicode Line Breaking Algorithm	⚠️ implemented in icu::segmenter, outdated
UAX 15	Unicode Normalization Forms	✅ implemented in icu::normalizer
UTS 18	Unicode Regular Expressions	⚠️ level 1 implemented in regex, cannot use icu::properties data
UAX 24	Unicode Script Property	✅ implemented in icu::properties. ⚠️ implemented in unicode-script, cannot use icu::properties data, limited interop
UAX 29	Unicode Text Segmentation	⚠️ implemented in icu::segmenter, outdated
UAX 31	Unicode Identifiers and Syntax	❌ partial implementation in unicode-xid and unicode-script, cannot use icu::properties data
UAX 34	Unicode Named Character Sequences	❌ not implemented, data not in icu::properties
UTS 35	Unicode Locale Data Markup Language (LDML)	⚠️ partially implemented in icu::calendar, icu::datetime, icu::decimal, icu::list, icu::locale, icu::pattern, icu::plurals, icu::time
UTR 36	Unicode Security Considerations	probably superseded by UAX 31, UAX 39, UAX 55
UTS 37	Unicode Ideographic Variation Database	does not specify algorithms
UAX 38	Unicode Han Database (Unihan)	does not specify algorithms
UTS 39	Unicode Security Mechanisms	⚠️ partially implemented in unicode-security, cannot use icu::properties data
UAX 41	Common References for Unicode Standard Annexes	does not specify algorithms
UAX 42	Unicode Character Database in XML	does not specify algorithms
UAX 44	Unicode Character Database	does not specify algorithms
UAX 45	U-Source Ideographs	does not specify algorithms
UTS 46	Unicode IDNA Compatibility Processing	✅ implemented in idna, uses icu
UAX 50	Unicode Vertical Text Layout	⚠️ implemented in harfbuzz, however the relevant properties are not used through harfbuzz-traits (e.g. icu) ⚠️ implemented in harfrust, does not support external Unicode data sources at all
UTS 51	Unicode Emoji	⚠️ partially implemented in icu::properties
UAX 53	Unicode Arabic Mark Rendering	⚠️ implemented in harfbuzz, however the relevant properties are not used through harfbuzz-traits (e.g. icu) ⚠️ implemented in harfrust, does not support external Unicode data sources at all
UTS 55	Unicode Source Code Handling	❌ not implemented
UAX 57	Unicode Egyptian Hieroglyph Database (Unikemet)	does not specify algorithms
UTS 58	Draft Unicode Link Detection and Serialization	❌ not implemented
UTR 59	Proposed Draft East Asian Spacing	❌ not implemented
UAX 60	Draft Data for Non Han Ideographic Scripts	does not specify algorithms
UTS 61	Proposed Draft Unicode Set Notation	⚠️ partial experimental support, missing some data such as blocks and character names

[Manishearth]

• @hsivonen Fairly recently changed gecko vendoring to block vendoring of crates that duplicate icu4x. Learned that there are multiple unicode crates that have perf claims (like unicode-ident). And unicode-ident has perf claims based on Raph's algorithms. For some of these things it makes sense to have a CPT-like bitmap with packed 1 bit data instead of an inversion list. We should investigate 1-bit, 2-bit, 4-bit, 24-bit extensions to CPT. The fast vs small runtime thing cannot be good for icu_properties.
• @Manishearth note unicode-ident/unicode-xid cannot use icu4x because of cyclic deps.
• @Manishearth Three big questions
  • Which algorithms should we be supporting?
  • Should we convince unicode-rs crates to use ICU4X
  • What's the situation perf-wise/etc
• @hsivonen The thing that triggered my attempts to block introduction of these crates was unicode-normalization being introduced to gecko. If we have this issue of cyclic deps, one possibility
• @hsivonen Worth exploring an ICU4X cargo feature that delegates the same algorithms to ICU4X data. Currently icu_normalizer API surface allows for faster things. str->cow is faster than unicode-normalization with collect.
• @Manishearth So these crates are a hodgepodge, not a single unified story. Different maintenance schedules, etc. Also people care about small crates here. So not really in favor of moving unicode-rs to using icu4x, better to have people explicitly move over.
• @hsivonen Seel also: Should we port Firefox's IDNA policy function to Rust and then do the provenance work to relicense to something other than MPL.
• @hsivonen at least one TLD that wants to support domains in a restricted-use script.
• @sffc Could basically adopt the unicode-rs crates. I think this question will come up over and over again.
• @Manishearth Because of the haphazard maintenance story of these crates I'd rather not spend too much effort working on them. I'd rather make ICU4X good and keep it maintained and have people switch over to it over time.
• @hsivonen Note also: the unicase crate that is a remaining duplicative crate in Gecko at the moment.
• @zbraniecki Would like to talk about another area with duplication. There's a unic_locale in Firefox that duplicates ICU4X locale. Fluent langneg duplicates ICU4X locale negotiation. Once I'm done with locale neg I'd like to move Fluent over to ICU4X.
• @sffc Note that these crates are just better in some ways, they're faster and smaller
• @Manishearth: highlights that these crates are not a monolith. their properties vary
• @sffc One possible outcome is that we make these good and make them the canonical place
• @sffc other is icu_properties becomes the canonical home
• @hsivonen We have strategies to improve icu_properties; should we try doing that more? We should do actual benchmarks. Compare the size and speed compared to Raph's impl. If it's still slower we can try other paths.
• ...
• @Manishearth not sure more discussion is productive here but we should asynchronously talk about optimizations and such
• @sffc Focus on the specific situation at hand, linebender. They tried using icu_properties and it's slower than handwritten tables. Ideally they would use ICU4X props and not duplicate the data.
• @hsivonen since linebender doesn't need backcompat with previously generated serde loadables, should be possible to put properties behind an enum with the most efficient variant
• @sffc My understanding is that some of the linebender props can be written as three if statements in code, which will always be faster than a CPT.
• @sffc something to be said for if it's not good for linebender to use CPT then it isn't for ICU4X. The advantage of CPT is that we can unify the data generation, but if there's an alt solution that's significantly faster...it would be good for ICU4X's users too
• @hsivonen which props?
• @hsivonen If we do my bit-CPT optimization then it should be as fast as 3 if statements. CPT when used in the way it's designed to be used in ICU4C is quite fast: you use one branch to find if you're in the BMP or not and then after that there are no branches. So it's really nice perf wise, if not size wise.
• @Manishearth agree that we should try and pick up perf improvements
• @sffc n.b. "3 if statements" will still be faster than CPT because of the lack of loads
• @hsivonen We should try the CPT optimizations
• @Manishearth n.b. we could optimize Property::for_char without changing backing storage

(break)

• @sffc glad to hear that we seem to be in alignment on making icu_properties faster where possible. Good first step to try some of these if statement optimizations in for_char.
• @sffc there are a few normalizer props because we pack them in
• @sffc do we want to pack the linebender props?
• @Manishearth looking at the bidi stuff we have packed bidi mirroring data instead of regular props because you'd never want these individually
• @Manishearth but we could have PackedScriptRenderingInfo as a thing that you're just expecte dto not mix with data loads of its subproperties
• @Manishearth Also could even teach baked data to load from the same packed type. Similar to what we're discussing for displaynames/units where we don't use a regular locale/attribute-to-value CPT
• @Manishearth e.g. CodePointSetData gets a variant that references a trie and some sort of slicing index (or something)
• @Manishearth Next steps would be to
  • Understand what ICU4C props are packed
  • Understand how they are packed, if it's a CPT or something else
  • Understand what linebender needs
  • Understand linebender's optimizations (are they better than using ICU4C's optimizations?)
  • Measure perf/etc.
• @sffc worried about how this works with the traits
• @Manishearth So https://unicode-org.github.io/icu4x/rustdoc/icu/properties/provider/enum.PropertyCodePointSet.html would get a new variant, Packed(SomePackedType, SlicingInfo) where slicing info tells you how to get the sub-data out of it. Baked data would know to generate a proxy load, this is just datagenned into baked data. The hard part is buffer, we'd need to include additional information about packing on the DataMarkerInfo.

[sffc]

I'm triaging this as Backlog because it's not clear what the action is. If there is a specific action, we can change the milestone.