[API Proposal]: `Base64(Url).EncodeToXXX(ReadOnlySequence<byte>)`

[RamType0]

Background and motivation

I want to implement ZLib -> Base64Url encode to bytes.

1. Use ZLibStream for compression
2. Use RecyclableMemoryStream as the stream to which compressed data is written because I wanna reduce allocations.
3. Get compressed data by RecyclableMemoryStream.GetReadOnlySequence().

Then, how could we encode it to Base64 URL?

You may think that this would be work.

Span<char> encodedChars = stackalloc char[Base64Url.GetEncodedLength((int)sequence.Length)];
var destination = encodedChars;
var remainingLength = sequence.Length;
foreach (var memory in sequence)
{
    var span = memory.Span;
    remainingLength -= span.Length;
    Base64Url.EncodeToChars(span, destination, out var bytesConsumed, out var charsWritten, remainingLength == 0);
    destination = destination[charsWritten..];
}

Actually, we need to process border of segments.

API Proposal

namespace System.Buffers.Text

public static class Base64
{
    public static int EncodeToUtf8(ReadOnlySequence<byte> source, Span<byte> destination); 
    public static int EncodeToChars(ReadOnlySequence<byte> source, Span<char> destination); 

    public static void EncodeToUtf8(ReadOnlySequence<byte> source, IBufferWriter<byte> destination); 

    // This one could be not needed as IBufferWriter<char> is not commonly used
    public static void EncodeToChars(ReadOnlySequence<byte> source, IBufferWriter<char> destination); 

    public static string EncodeToString(ReadOnlySequence<byte> source); 
}
public static class Base64Url
{
    public static int EncodeToUtf8(ReadOnlySequence<byte> source, Span<byte> destination); 
    public static int EncodeToChars(ReadOnlySequence<byte> source, Span<char> destination); 

    public static void EncodeToUtf8(ReadOnlySequence<byte> source, IBufferWriter<byte> destination); 

    // This one could be not needed as IBufferWriter<char> is not commonly used
    public static void EncodeToChars(ReadOnlySequence<byte> source, IBufferWriter<char> destination); 

    public static string EncodeToString(ReadOnlySequence<byte> source); 
}

API Usage

using RecyclableMemoryStream memoryStream = MemoryStreamManager.GetStream();


using (ZLibStream zlibStream = new(memoryStream, compressionLevel, leaveOpen: true))
{
    zlibStream.Write(utf8DiagramSource);
}

var compressedSequence = memoryStream.GetReadOnlySequence();

return Base64Url.EncodeToString(compressedSequence);

Alternative Designs

I have created draft implementation.

public static int EncodeToChars(ReadOnlySequence<byte> source, Span<char> destination)
{
    const int BlockSize = 3;

    Span<byte> processBorderBuffer = stackalloc byte[BlockSize];

    var reader = new SequenceReader<byte>(source);
    var totalCharsWritten = 0;
    while (true)
    {
        var operationStatus = Base64Url.EncodeToChars(reader.UnreadSpan, destination, out var bytesConsumed, out var charsWritten, reader.UnreadSequence.IsSingleSegment);
        Debug.Assert(operationStatus is OperationStatus.NeedMoreData or OperationStatus.Done);

        reader.Advance(bytesConsumed);
        destination = destination[charsWritten..];
        totalCharsWritten += charsWritten;

        if (reader.End)
        {
            break;
        }

        if (reader.UnreadSpan.Length < BlockSize)
        {
            // Operation has been interrupted without completely consumed current span.
            // Need to process border.

            if (reader.TryCopyTo(processBorderBuffer))
            {
                reader.Advance(BlockSize);
            }
            else
            {
                // Remaining data is less than 3
                // UnreadSpan has at least 1 length and there is next segment with at least 1 length
                // So remaining data must be 2 bytes

                // No longer to maintain buffer, operation is almost over
                processBorderBuffer = processBorderBuffer[..2];

                var result = reader.TryCopyTo(processBorderBuffer);

                Debug.Assert(result);

                reader.AdvanceToEnd();
            }

            operationStatus = Base64Url.EncodeToChars(processBorderBuffer, destination, out bytesConsumed, out charsWritten, reader.End);

            Debug.Assert(operationStatus is OperationStatus.NeedMoreData or OperationStatus.Done);

            destination = destination[charsWritten..];
            totalCharsWritten += charsWritten;

            if (reader.End)
            {
                break;
            }

        }

    }
    return totalCharsWritten;
}

Risks

No response

[huoyaoyuan]

This is not currently actionable since ReadOnlySequence lives in a higher level of assembly.

The problem isn't specific to Base64Url, but all the span-encoding APIs. There needs to be an helper to handle the borders.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-runtime
See info in area-owners.md if you want to be subscribed.

[RamType0]

The problem isn't specific to Base64Url, but all the span-encoding APIs. There needs to be an helper to handle the borders.

Yes, I think so.
This is just an a part of, or the example of the problem.

This is not currently actionable since ReadOnlySequence lives in a higher level of assembly.

What do you mean?

[huoyaoyuan]

Base64Url lives in CoreLib (where object lives), it's the lowest level library and can only reference things from itself. ReadOnlySequence lives in System.Memory. It can't be referenced by Base64Url.

[RamType0]

How about using this feature?

https://learn.microsoft.com/dotnet/csharp/whats-new/csharp-14#extension-members

[huoyaoyuan]

It's totally unrelated. Base64Url is a static class. No matter whether we make the methods appears to be attached with Base64Url, we still need to explicitly define where its implementation lives. Extension methods are always reviewed in non-extension form.

[RamType0]

I meant we can define overloads for ReadOnlySequence<byte> out side of CoreLib with this feature.

[RamType0]

Also, what I posted is not a well-known extension method feature.

It is a new feature in C#14, and we can add static methods to class with this feature

[huoyaoyuan]

Also, what I posted is not a well-known extension method feature.

It's very well-known to me and the maintainers before C# 14 starts.

I meant we can define overloads for ReadOnlySequence<byte> out side of CoreLib with this feature.

Yes, but where to define it needs to be explicitly designed.

And also, a general solution will be much preferred, instead of supporting Base64Url only.

[gfoidl]

@RamType0 as a shameless self-promotion have a look at my Base64 lib (NuGet). It supports ReadOnlySequence<byte> and IBufferWriter<byte>.

A code path for AVX-512 is missing at the moment.