Better Unicode versions support

[jerch]

Tracking issue for better Unicode version support.

Things to do:

• create a build tool based on https://github.com/fluffos/fluffos/tree/master/src/thirdparty/widecharwidth to extract needed Unicode information for a specific version
• automate version addon creation as far as possible with the build tool
• options to treat ambiguous character as double width

better Unicode support in general:

• bidi and grapheme support
• long run: conceptual work towards a better unicode handling in terminals via terminal-wg

[thefallentree]

just reminder that we also need have an options to treat ambiguous character to double width

[jerch]

@thefallentree Woops, that was the initial reason to create the issue, thx for pointing out (moved it up).

[gustavopch]

Does this issue address overlapping emoji problems in xterm terminals like the one in the following screenshot taken from VSCode integrated terminal?

OS: Arch Linux 5.11.2-arch1-1
VSCode: 1.53.2
Emoji font: Noto Color Emoji (noto-fonts-emoji package from AUR)

[jerch]

@gustavopch Yeah ideally most of these issue would be handled by that. Still there are corner cases where a proper result cannot be determined - if app side and the terminal host system have different ideas about a certain codepoint width we have to prioritize one side. Here we would have to go with app side by default to not get out of sync with line length assumption of the app.

This is mostly related to different unicode versions being used on either side and currently there is simply no way to harmonize app and terminal side in this regard. Not sure if we will ever see a good solution to that (would need a really advanced unicode terminal interface), also note that most local terminals do not suffer as much as xterm.js does (due its decoupled terminalview <--> PTY host nature).

[thefallentree]

I've finally done the leg work of making a widechar_wcwidth() JS version, directly generated from newly released unicode 14

see
https://github.com/fluffos/fluffos/blob/master/src/www/widechar_width.js

and see my xtermjs unicode 14 provider

https://github.com/fluffos/fluffos/blob/master/src/www/xterm-addon-unicode14.js

[jerch]

@thefallentree 👍 Wow thats nice. Care to setup a build script we can use in xterm.js repo, that grabs the width definitions from the official unicode resources?

For v13 and v14 support I think we should get the grapheme clustering done in xterm.js. For that we'd need an extension of the unicode provider to also provide the segmentation tables and logic. So if you want to work on that - feel free to create a PR.