Should wcwidth have "Treat ambiguos-width as wide" option?

[keatonLiu]

import wcwidth

if __name__ == '__main__':
    print(wcwidth.wcswidth("①你好"))
    print(wcwidth.wcswidth("你好啊"))

results in:

But it displays 2 character width in monospace font:

[jquast]

Which terminal emulator are you using in this example?

For iTerm2, this is correct,

As well as WezTerm,

And also Kitty,

[keatonLiu]

Maybe wcwidth only focus on terminal font? I'm using PrettyTable to generate table, which depends on wcwidth, and I want to display the table text on browser. For example, I'm using chrome, and I found monospace fonts works fine most of the time. But for some unicode words, it displays with a different length.

[keatonLiu]

It will be helpful if I can provide the font family and get a more general result. Is it possible?

[jquast]

wcwidth is primarily focused for terminals, that is if browsers and terminals disagree we would rather match with terminals. Although I expect a javascript or browser-based library that is more focused on browser width, I cannot find one at this moment, please suggest if you do.

Browsers are able to communicate directly with the font engine of the operating system, while wcwidth in python and other languages are not, so we generally take a more naive approach. And this is probably why most terminals are also wrong in this case while browsers are not.

In this case, the problem with ① (https://codepoints.net/U+2460) is that it is Ambiguous width (https://unicode.org/reports/tr11/#Ambiguous) and,

They have a “resolved” width of either narrow or wide depending on the context of their use.

In the following code blocks I use the same character, one with english letters on the same line,

①2345
12345

and another of your example with your Mandarin Chinese "hello",

①你好
12345

Although they render differently sized, at least on my browser (Firefox 120.0.1), they have approximately the same width. I will say that monospace fonts do not always align vertically in browsers (note how the number '5' does not align in the first example), while they always do in terminals.

Screenshot of the above,

(End screenshot)

It would require more experimentation, but maybe for a page of Chinese locale it would render differently, such as in your original screenshot, I'm not really sure.

In any case, there are options on many terminals, to cause ambiguous width characters to display as 2 cells,

• Wezterm https://wezfurlong.org/wezterm/config/lua/config/treat_east_asian_ambiguous_width_as_wide.html?h=ambiguous
• iTerm2 section ("Treat ambiguos-width characters as double width") https://iterm2.com/3.2/documentation-preferences.html

I'm not certain, but maybe this option is more frequently used for east-asian language users in terminals?

But it is very problematic -- the entire software stack needs to agree to "treat ambiguous width as wide", for example, here is an "$LD_PRELOAD-able library and a wrapper script" that patches posix wcwidth for this option, and references many issues and bugs about this option. https://github.com/fumiyas/wcwidth-cjk

The "Terminal Working Group" tried to come to a consensus about this and other issues, https://gitlab.freedesktop.org/terminal-wg/specifications/-/issues/9#note_406682 -- there was a great deal of discussion but this "Working Group" specifications project has failed to come to any consensus at all on any single issue (the "accepted" folder is empty, 31 open issues)

And, maybe this library could also provide such an option, to "treat ambiguous width as wide". And, I will rewrite this github issue to match that request.

[GalaxySnail]

It's also rendered with a width of 1 in Windows Terminal.

Even more, it's rendered with a width of 1 in my webbrowser (chromium).

I personally agree that "①" should be East Asian Wide, but unfortunately it is East Asian Ambiguous (and a similar character U+2780 is East Asian Neutral). In my opinion, it may need to be addressed in Unicode, but I'm not sure. Unicode is a bit chaotic. ¯\_(ツ)_/¯

[keatonLiu]

Thank you for so much work! You are very helpful.
I have tested that in my windows terminal and gets the same result.

I understand it is because the ① character is an East Asian Ambiguous character, which is treated as different size in different context. I agree that it can have a "treat ambiguous width as wide" option because in most cases it displays the same size as a east asian character in my locale.
You can visit this website and get an intuitive demo: https://www.zhonghuazidian.com/zi/%E2%91%A0
On my browser, chrome:

Even in Word:

I think it will be a wide width character if you use a monospace font-family in browser.

[keatonLiu]

wcwidth is primarily focused for terminals, that is if browsers and terminals disagree we would rather match with terminals. Although I expect a javascript or browser-based library that is more focused on browser width, I cannot find one at this moment, please suggest if you do.

Browsers are able to communicate directly with the font engine of the operating system, while wcwidth in python and other languages are not, so we generally take a more naive approach. And this is probably why most terminals are also wrong in this case while browsers are not.

In this case, the problem with ① (https://codepoints.net/U+2460) is that it is Ambiguous width (https://unicode.org/reports/tr11/#Ambiguous) and,

They have a “resolved” width of either narrow or wide depending on the context of their use.

In the following code blocks I use the same character, one with english letters on the same line,

①2345
12345

and another of your example with your Mandarin Chinese "hello",

①你好
12345

Although they render differently sized, at least on my browser (Firefox 120.0.1), they have approximately the same width. I will say that monospace fonts do not always align vertically in browsers (note how the number '5' does not align in the first example), while they always do in terminals.

Screenshot of the above, (End screenshot)

It would require more experimentation, but maybe for a page of Chinese locale it would render differently, such as in your original screenshot, I'm not really sure.

In any case, there are options on many terminals, to cause ambiguous width characters to display as 2 cells,

• Wezterm https://wezfurlong.org/wezterm/config/lua/config/treat_east_asian_ambiguous_width_as_wide.html?h=ambiguous
• iTerm2 section ("Treat ambiguos-width characters as double width") https://iterm2.com/3.2/documentation-preferences.html

I'm not certain, but maybe this option is more frequently used for east-asian language users in terminals?

But it is very problematic -- the entire software stack needs to agree to "treat ambiguous width as wide", for example, here is an "$LD_PRELOAD-able library and a wrapper script" that patches posix wcwidth for this option, and references many issues and bugs about this option. https://github.com/fumiyas/wcwidth-cjk

The "Terminal Working Group" tried to come to a consensus about this and other issues, https://gitlab.freedesktop.org/terminal-wg/specifications/-/issues/9#note_406682 -- there was a great deal of discussion but this "Working Group" specifications project has failed to come to any consensus at all on any single issue (the "accepted" folder is empty, 31 open issues)

And, maybe this library could also provide such an option, to "treat ambiguous width as wide". And, I will rewrite this github issue to match that request.

Interesting, I'm using chrome and displays in another way:

[fancidev]

I ran into a similar issue when displaying a Unicode filled square character U+25A0 in Windows console (command line prompt) using python_promptkit. Under one font it renders as single width; under another font it renders as double width.

Since Windows doesn’t provide a wcwidth function, it seems quite hard to tell the rendered width reliably. Maybe prompt_toolkit could actually render the text and compute the width from cursor advancement, but that seems a bit too much. So I just replace the Unicode character with an ASCII character that is definitely single width…

[jquast]

Thanks for chiming in @fancidev, I did actually write a tool that does just as you describe, "could render the text and compute the width from cursor advancement", that is definitely possible! https://github.com/jquast/ucs-detect

Maybe there could be an environment variable that ucs-detect could export that wcwidth could make use of to more accurately determine any widths outside of specification of a given terminal

[fancidev]

Thanks for the info. Good to know there is already such a facility!

The demo on the homepage of ucs-detect runs through the characters on screen. I wonder if the width detection can be performed without echo?

If that’s possible, a possible solution could be to run through the ambiguous characters on application start-up (as well as upon special console events such as front change) and remember the results. (If the delay is small, the end-user would not feel it.)

[jquast]

Yes, it is possible. At least, you can determine the background color of a "smart" terminal, https://github.com/jquast/blessed/blob/6181f5ab2e3682d997a96030a3b72c09346e72f5/blessed/terminal.py#L710-L729 and fallback to assuming black, and you can set the foreground color to be equal. And of course, temporarily disable visible cursor.

I'll try to work towards updating ucs-detect with the sort of technique you look for, to detect ambigous width setting, and something like "--hidden" with the suggested technique. The more advanced language tests I prefer to be multi-line and visible, it helps me see artifacts such as which terminal emulators are slow, or 100x slower on certain language tests

[GalaxySnail]

I personally agree that "①" should be East Asian Wide, but unfortunately it is East Asian Ambiguous (and a similar character U+2780 is East Asian Neutral). In my opinion, it may need to be addressed in Unicode, but I'm not sure. Unicode is a bit chaotic. ¯_(ツ)_/¯

FWIW, Unicode 16 introduced 8 standardized variation sequences for curly quotation marks (‘ ’ “ ”) to select halfwidth or fullwidth variants. They aren't useful for most East Asian Ambiguous characters (like "①"), but it's better than nothing.

• https://www.unicode.org/versions/Unicode16.0.0/#:~:text=Eight%20variation%20sequences%20have%20been%20added%20for%20curly%20quotation%20marks
• https://www.unicode.org/Public/16.0.0/ucd/StandardizedVariants.txt

# East Asian punctuation positional and width variants

2018 FE00; non-fullwidth form; # LEFT SINGLE QUOTATION MARK
2018 FE01; right-justified fullwidth form; # LEFT SINGLE QUOTATION MARK
2019 FE00; non-fullwidth form; # RIGHT SINGLE QUOTATION MARK
2019 FE01; left-justified fullwidth form; # RIGHT SINGLE QUOTATION MARK
201C FE00; non-fullwidth form; # LEFT DOUBLE QUOTATION MARK
201C FE01; right-justified fullwidth form; # LEFT DOUBLE QUOTATION MARK
201D FE00; non-fullwidth form; # RIGHT DOUBLE QUOTATION MARK
201D FE01; left-justified fullwidth form; # RIGHT DOUBLE QUOTATION MARK