Emoji: How to measure Non-Recommended Emoji sequences?

[janlelis]

Hi Jeff,

imho, one of the many problems of getting Emoji widths right is that there are multiple definitions of what an Emoji sequence can look like to be considered an Emoji. The standard defines an recommended set here:

https://www.unicode.org/reports/tr51/#def_rgi_set

but it also defines variations of it (in emoji-test.txt, listed here) and more importantly, allows arbitrary Emoji sequences which are still considered valid.

However, probability is low that non-recommended Emoji will ever gain so much popularity that terminals would need to display them as a single width-2-Emoji, so there will always be the case, that some Emoji sequences should be displayed as an (actual) sequence of separate basic Emoji. That is the reason I made Emoji handling configurable in the latest release of unicode-display_width.

I did some manual testing and was surprised to learn that most terminals (gnome-terminal, vscode terminal etc.) would not display most Emoji sequences (the exception being popular terminals on macOS). I also noted that is very unpleasant to work with software that measures RGI-Emoji sequences correctly, but the terminal displays them as separate Emoji. I am trying to automatically detect the terminal used to provide a good out-of-box experience, but probably, a run-time check (like ucs-detect does) would be more reliable.

Condensing the above remarks into a single question/issue, it would be:

Should wcwidth() / the wcwidth spec support different string width mechanisms for RGI and non-RGI Emoji?

[jquast]

I totally understand and agree, the results of my ucs-detect library so far (and preliminary results from a more detailed WIP branch) show terminal support varies more wildly than expected.

I expected each terminal to mostly match a specific unicode release, and implemented UNICODE_VERSION for that. But there are countless examples where Emoji, Zero-width, and combining character tables of a terminal are incomplete and the unicode versions of each table are different!

I have proposed a complex solution in another issue, here, #123 (comment)

Maybe there could be an environment variable that ucs-detect could export that wcwidth could make use of to more accurately determine any widths outside of specification of a given terminal

And the very bottom of this issue, #104 (comment)

However, with tools like 'ucs-detect', we can very programmatically determine with black-box testing, which wide, zero-width, and whether ZWJ and VS-16 are supported, right down to exactly which ones. By making this a delta of expected terminal support, and using ranges with codepoints, maybe it is possible to describe with a complex environment variable.

Just spitballing an idea of what it might look like,

UNICODE_SUPPORT="zero[8.0:!category:Mc,Mn,!1001-1002,!1003],wide[15.1:!zwj,!vs16,!9009-9010]"

It's pretty advanced, however, this would allow us to precisely predict or measure the width of all possible sequences on each individual terminal with more perfect accuracy.