Utf8.TryWrite applies alignment by counting bytes instead of characters

[jdryden572]

Description

When using Utf8.TryWrite to write an interpolated string as UTF8 bytes, passing an alignment value with any of the formatted values does not always result in the same amount of padding as when using string.Format or default string interpolation. If the formatted value has any non-ASCII characters then then less padding will be added.

Reproduction Steps

using System;
using System.Text;
using System.Text.Unicode;

string[] examples = new[]
{
    "\u0108",       // Ĉ 1 char, 2 bytes UTF8
    "\u20ac",       // € 1 char, 3 bytes UTF8
    "\ud83d\ude00", // 😀 2 chars, 4 bytes UTF8
};

foreach (string s in examples)
{
    Console.WriteLine($"utf16: [{s,4}]");
}
foreach (string s in examples)
{
    Span<byte> span = new byte[8];
    Utf8.TryWrite(span, $"[{s,4}]", out int written);
    Console.WriteLine("utf8:  " + Encoding.UTF8.GetString(span.Slice(0, written)));
}

Expected behavior

Formatting a value with an alignment in Utf8.TryWrite should produce the same amount of padding in UTF8 as is added in other .NET string (UTF16) formatted strings.

For the code snippet above, it should produce:

utf16: [   Ĉ]
utf16: [   €]
utf16: [  😀]
utf8:  [   Ĉ]
utf8:  [   €]
utf8:  [  😀]

Actual behavior

When the formatted value includes any characters that require more than 1 byte in UTF8 encoding, the alignment is incorrect and produces less padding in Utf8.TryWrite.

For the code snippet above, it produces:

utf16: [   Ĉ]
utf16: [   €]
utf16: [  😀]
utf8:  [  Ĉ]
utf8:  [ €]
utf8:  [😀]

Regression?

This has been the behavior since the Utf8.TryWrite API was introduced in .NET 8, and it is also reproducible in .NET 9.

Known Workarounds

If the correct padding is really needed, default string interpolation or formatting can be used to format the value as UTF16 in a string or a Span<char>, and then that UTF16 can be encoded into the UTF8 output Span<byte> using Encoding.UTF8.GetBytes.

This loses the nice ergonomics of formatting directly into the UTF8 buffer, and either allocates (if making a string) or requires more buffer management to get a Span<char>.

Configuration

Tested in .NET 8 & .NET 9 Preview
I'm using Windows x64, but I'm pretty sure this is not platform/arch dependent.

Other information

Before I begin -- I am interested in trying to fix this and I'm happy to open a PR for it. It would be my first time contributing however, so I understand if you feel someone else should handle fixing it instead.

The issue is that the amount of required padding is being determined by counting how many bytes were written, even though we're working with UTF8 where many characters take more than one byte. Here's the culprit:

runtime/src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8.cs

Line 778 in e133fe4

int paddingNeeded = alignment - bytesWritten;

The simple (and maybe too naiive) approach to fix this would be to use Encoding.UTF8.GetCharCount on the slice that was written, to measure how many chars the formatted text ended up writing. But this private method is called by multiple overloads of AppendFormatted and for some of them, we alread know how many chars we wrote. For example, if the value being formatted is a ReadOnlySpan<char> or string, we know how many chars it had. Or if it was ISpanFormattable, we already formatted it into our own Span<char> buffer before writing and know how many chars there are.

So I think a better solution might be to find a way to have the overloads pass an optional int charsWritten if they know how many there were. If not, the alignment handling should call Encoding.UTF8.GetCharCount on the bytes we wrote so far to calculate how many chars it ended up being.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-text-encoding
See info in area-owners.md if you want to be subscribed.

[tarekgh]

CC @stephentoub

[tannergooding]

This isn't actually any different from string.Format, you can see the same occur if you use any surrogate characters or non-combining marks

Consider

Console.WriteLine($"{"A",5}");   // Length: 1
Console.WriteLine($"{"👨‍👦",5}");  // Length: 5

//    A
//👨‍👦

[tannergooding]

The main consideration is that it is easier to see such points of failure with UTF-8 because more things require multiple characters to represent.

Even with Rune where everything is functionally "1 character" you would have the same issue, because there is a distinction between "number of code points" and "amount of visual space taken". -- A code point is a singular Unicode value in the range 0 to 0x10FFFF, it may be represented by 1 or more characters depending on the encoding (UTF-8 is 1-4 characters per code point, UTF-16 is 1-2 characters per code point, UTF-32 (effectively Rune) is always 1 character per code point).

👨‍👦, for example, is 3 unicode code points represented by man + zero-width joiner + boy ("👨" + "‍" + "👦", where the middle one there is not an empty string, its an invisible code point that takes zero visual space).

[jdryden572]

Thanks for the reply, @tannergooding!

I agree that the notion of "alignment" in format strings is more complicated than simply counting visual characters, and your example is a very good one. However, I disagree with this statement:

This isn't actually any different from string.Format ...

Because the problem (as I see it) is that they are in fact producing something different. It's more apparent if we add enough padding to see at least one padding character in each encoding:

Span<byte> span = new byte[16];
Utf8.TryWrite(span, $"[{"👨‍",8}]", out int written);
Console.WriteLine(Encoding.UTF8.GetString(span.Slice(0, written)));
Console.WriteLine($"[{"👨‍",8}]");

// [ 👨‍]
// [     👨‍]

My hope was that Utf8.TryWrite would produce the same amount of padding as string.Format, even if the latter has its own quirks regarding alignment on values that cannot be represented in one UTF16 character.

[elgonzo]

This issue is not just about surrogate pairs. Surrogate pairs are by definition not a single code point. But two of the character examples given in the issue report are single code points, not pairs of code points or pairs of characters with a combining mark or some such.

While i can (somewhat) understand the behavior regarding surrogate pairs and any sequence of two or more code points that renders as a single character, what exactly is the useful purpose of a padding algorithm which makes its behavior dependent on the bit-length/number of the underlying code units (octets) for a single code point? It's string interpolation and not byte buffer interpolation, after all.

[tannergooding]

Edit: Adjusted to use the term "scalar value" which is more appropriate than code point.

Because the problem (as I see it) is that they are in fact producing something different

They are producing different outputs, but still behaving identically. There is a difference in the concepts.

The behavior for alignment is that it is in respect to the number of characters. UTF-8, UTF-16, and UTF-32 use a different number of characters to represent the same data; thus an alignment of 5 with a UTF-8 string is going to often be different from an alignment of 5 for a UTF-16 string, because the length of each differs even if they appear visually identical.

Surrogate pairs are by definition not a single code poi

This is incorrect, what you're thinking of is Unicode Characters. As per https://unicode.org/glossary/

• Character. (3) The basic unit of encoding for the Unicode character encoding
• Code Point. (1) Any value in the Unicode codespace; that is, the range of integers from 0 to 0x10FFFF.
• UTF-8. A multibyte encoding for text that represents each Unicode character with 1 to 4 bytes
• UTF-16. A multibyte encoding for text that represents each Unicode character with 2 or 4 bytes

The terminology that various programming languages use and that occurs in common conversation tends to be much more loose, with things getting mixed up and used interchangeably. What is formally defined by the Unicode spec here, and in terms of how .NET operates is much more strict.

.NET strings have never worked on code points scalar values as fundamental units; they have always been a representation of individual characters. This in turn shows up as part of indexing, default comparisons, and so on. The use of UTF-16 by default and most typical text not requiring multiple characters or code points to represent a single visual glyph has lead to incorrect presumptions (and often bugs) when such text is encountered, particularly as emoji has become more commonplace.

[tannergooding]

There are in many cases overloads of functions, additional parameters you can pass in, or alternative APIs you can use that do try and operate in terms of more "visual" information or in terms of the abstract data that is represented; such that for example à (the individual code point 0x00E1) and à (the letter a, 0x0061, + the combining diacritical grave accent ◌̀, 0x02CB) are treated as eqaul.

But, such a scenario does not currently exist for string.Format and the alignment functionality that is available for it. Such a request would be new and would need to consider many things, including concepts like combining characters, zero width joiners, and similar.

[elgonzo]

Surrogate pairs are by definition not a single code point

This is incorrect

No, it's not incorrect. Excerpts from the same standard you linked to (https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-3):

Section 3.2.1

C1 A process shall not interpret a high-surrogate code point or a low-surrogate code point as an abstract character.
The high-surrogate and low-surrogate code points are designated for surrogate code units in the UTF-16 character encoding form. They are unassigned to any abstract character.

Section 3.4

D10a Code point type: Any of the seven fundamental classes of code points in the standard: Graphic, Format, Control, Private-Use, Surrogate, Noncharacter, Reserved.

Section 3.8

D71 High-surrogate code point: A Unicode code point in the range U+D800 to U+DBFF.

D73 Low-surrogate code point: A Unicode code point in the range U+DC00 to U+DFFF.