Grapheme rendering improvements

[akrantz]

When text is rendered, the font may actual draw the grapheme outside the number of cells reported. For my file viewer app using Terminal.Gui I've addressed this in two ways.

First I have a function GetGraphemeToDisplay(string grapheme) which can translate the string into what should be actually displayed. This is specific to my app but could be a useful general function provided by Terminal.Gui perhaps.

///

/// Returns the display form of a grapheme.
/// Transforms raw graphemes into what should actually be displayed:
/// - control chars → control picture symbols (␀, ␊, etc.) when showControlPictures is true
/// - combining marks → dotted circle + mark (◌̀)
/// - flags → country code + 🏳
/// - keycaps → [X]
/// - wide-rendering scripts → char + trailing space
/// - zero-width/invisible → empty
/// - lone surrogates, unassigned, replacement char → replacement char or empty
/// - everything else → itself at measured width
///

public (string displayText, int cellWidth) GetGraphemeToDisplay(string grapheme, bool showControlPictures = false, bool showReplacementChar = false, bool showWhitespace = false)

This uses another function which reports the grapheme width in cell units. This is to address an issue that is seen in how the underlying grapheme width is reported, because what you see is that for East Asian scripts, the reported width is not really wide enough, so there is padding needed. This is based on Windows Terminal, and I'm not sure how well this maps across different platforms, fonts, etc. I just adjusted this manually looking at some examples but haven't done this completely. I wanted to discuss if it would be helpful to bring either of these over into Terminal.Gui.

private static int GetCodepointPadding(int codepoint) => codepoint switch
{
    // Arabic — wide-base forms (beh-like, seen-like, sad-like, feh-like)
    0x0628 => 1,
    >= 0x062A and <= 0x062B => 1,
    >= 0x0633 and <= 0x0636 => 1,
    0x0641 => 1,
    0x066E => 1,
    >= 0x0679 and <= 0x0680 => 1,
    >= 0x069A and <= 0x069E => 1,
    >= 0x06A1 and <= 0x06A6 => 1,
    0x06AA => 1,
    >= 0x06FA and <= 0x06FB => 1,

    // Devanagari — select characters that render wide
    0x0904 => 1,
    >= 0x0905 and <= 0x0906 => 1,
    >= 0x0909 and <= 0x090B => 1,
    >= 0x0911 and <= 0x0916 => 1,
    0x0919 => 1,
    0x0950 => 1,
    0x095E => 1,
    0x0960 => 1,
    >= 0x0974 and <= 0x0975 => 1,

    // Bengali (U+0980–U+09FF) — TODO: audit per-character
    >= 0x0980 and <= 0x09FF => 1,

    // Gurmukhi (U+0A00–U+0A7F) — TODO: audit per-character
    >= 0x0A00 and <= 0x0A7F => 1,

    // Gujarati (U+0A80–U+0AFF) — TODO: audit per-character
    >= 0x0A80 and <= 0x0AFF => 1,

    // Oriya (U+0B00–U+0B7F) — TODO: audit per-character
    >= 0x0B00 and <= 0x0B7F => 1,

    // Tamil — most letters render at 2 cells
    0x0B94 => 2,
    0x0BA3 => 2,
    0x0BB0 => 0,
    0x0BB9 => 2,
    >= 0x0B80 and <= 0x0BFF => 1,

    // Telugu (U+0C00–U+0C7F) — TODO: audit per-character
    >= 0x0C00 and <= 0x0C7F => 1,

    // Kannada (U+0C80–U+0CFF) — TODO: audit per-character
    >= 0x0C80 and <= 0x0CFF => 1,

    // Malayalam — most letters render at 2 cells, some at 1, some at 3
    0x0D05 => 2,
    0x0D06 => 2,
    0x0D07 => 0,
    0x0D08 => 2,
    0x0D09 => 0,
    0x0D0B => 0,
    0x0D12 => 0,
    0x0D1F => 0,
    0x0D10 => 2,
    0x0D14 => 2,
    0x0D1D => 2,
    >= 0x0D00 and <= 0x0D7F => 1,

    // Sinhala (U+0D80–U+0DFF) — TODO: audit per-character
    >= 0x0D80 and <= 0x0DFF => 1,

    // Myanmar (U+1000–U+109F) — TODO: audit per-character
    >= 0x1000 and <= 0x109F => 1,

    // Georgian (U+10D0–U+10FF) — TODO: audit per-character
    >= 0x10D0 and <= 0x10FF => 1,

    // Ethiopic (U+1200–U+137F) — TODO: audit per-character
    >= 0x1200 and <= 0x137F => 1,

    // Khmer (U+1780–U+17FF) — TODO: audit per-character
    >= 0x1780 and <= 0x17FF => 1,

    // Batak (U+1BC0–U+1BFF) — TODO: audit per-character
    >= 0x1BC0 and <= 0x1BFF => 1,

    _ => 0,
};

[BDisp]

I think it would be interesting to create a WcwidthExtensions class to fill in the remaining size discrepancies. But it would be advisable to create unit tests for these same graphemes that confirm they return a different size than the Wcwidth NuGet package, in case an update corrects these differences, and thus remove them from the code in the WcwidthExtensions class.
Or simply add them to the existing StringsExtensions class.