Character names

[js-choi]

Hey, I just spoke with @sffc at an Ecma TC39 meeting, and he referred me to here.

ICU4X will eventually need to support lookup of Unicode character names (the Name property of code points, as well as character name aliases and named character sequences).

I’ve been working on trying to compress the collection of Unicode character names into a succinct data structure, occupying as little space as possible (my goal is 100–200 kB) while maintaining efficient bidirectional random access. To that end, in my spare time I’ve been working on an (unfinished) article describing the data structure, as well as an (unfinished) reference JavaScript implementation.

I don’t anticipate my work being done until several months from now at the earliest. But @sffc tells me that work on character names in ICU4X hasn’t started yet either. When ICU4X does reach the point when character names start to get implemented, I’d love to help out however I can. In the meantime, I will continue to work on the JavaScript library before porting it to Rust.

[sffc]

CC @Manishearth @iainireland

First we need to decide if we want unames in ICU4X, and if so, we should consider the work of @js-choi. Putting on the backlog for now.

[js-choi]

Note that the article and the proof of concept are still unfinished (even though they are public). I can try to prioritize finishing them, once this topic gets close to leaving the backlog. Just let me know here when that happens.

But, even though they are still unfinished, it is already clear that there is a lot of low-hanging fruit to compress all Unicode names, while maintaining fast random bidirectional lookup, using succinct or near-succinct data structures.

[markusicu]

a) Getting Unicode character names can be useful for outputting debug info or data files.

b) Unicode character names as input I find not very useful. For example, UTS 18 suggests \N{character name} for non-printable characters, but the name is generally so long that it would make the regex harder to read than using a simple code point escape, and a comment nearby, outside the regex pattern string. Same for use in other types of source code.

c) Character names data is annoyingly large. It sounds like you have better ideas for how to compress them than I had, but it's still a lot of data for marginal usefulness.

d) You can look at what I cooked up for ICU in 1999: https://github.com/unicode-org/icu/blob/main/tools/unicode/c/genprops/namespropsbuilder.cpp contains the builder code, and the file starts with a brief description of the data format. The data for Unicode 15 is 289kB. (It used to look much better in Unicode 3.0 days...) Code point to name is pretty fast, but the other way is a slow search.

[js-choi]

@markusicu:
Cheers and thanks for the feedback; I’m a big fan of your work in general.

I definitely agree that the data cost of names is annoyingly large. I, too, would not suggest seriously considering their inclusion in ICU before it is demonstrated that they may be compressed into a succinct data structure (efficient in both time and space). I’m still working on this.

With regards to the usefulness of character→name vs. name→character, my own experience comes from implementing many parsers whose grammars need to treat specific characters in special ways. (These parsers only sometimes use regular-expression syntax; oftentimes these are functional combinator parsers or the like, created from composed expression trees.) I have always found it irksome to embed code points directly in my code—I find them to essentially be magic numbers that hinder readability.

It’s certainly true that some names are long. (The shortest name, “Ox”, is two letters long but the longest name, “Box Drawings Light Diagonal Upper Centre to Middle Right and Middle Left to Lower Centre”, is 88 characters long.) But the average length of names (when pooling the 35,205 explicitly defined names from the current UCD data files) is 25.78 (e.g., “Domino Tile Vertical-06-06”). This isn’t that bad. Good code often tries to emphasize using self-descriptive variable/function names, and a 26-long name wouldn’t be too out of place of such code.

It’s also true that comments can alleviate code points’ magic-number inscrutability, but this is ironically at the cost of increased wordiness (now you have a code-point hex and a longish name) and locality (the explanation of the code point is removed from its original location in the source).

(My ulterior motive is eventually to make a proposal to TC39 to add named character literals to the JavaScript language. But I would not dare to do so until I can allay the engine vendors’ concerns of storage, memory, and retrieval time—hence this project.)

Funnily enough, I (and some other TC39 members I’ve spoken to) may be confident about the usefulness of name→character lookup…but we’ve been much less certain about the usefulness of character→name retrieval. The best use case that I could imagine for the latter was metaprogramming (e.g., JavaScript transpilers and code minifiers). But your point about debugging general text data for human display is well taken.

Your namespropsbuilder.cpp from 1999 is fascinating. The fact that its token-based compression was already able to get down to 289 kB for Unicode 15 is impressive. I think there’s a lot of other low-hanging fruit to squeeze it much more (while improving name→character time performance). I don’t expect my demo to be done soon, but I’ll follow up when I make some significant progress. Cheers!