Add TLS ClientHello inspection API

[DeagleGross]

New API lets a server pause its handshake at the ClientHello and read the TLS client hello as was received by OpenSSL, before any certificate or server-option decision.

1. TlsContext.EnableClientHelloInspection

namespace System.Net.Security;

public sealed partial class TlsContext : IDisposable
{
+ public bool EnableClientHelloInspection { get; set; }
}

Opt-in flag, set on the server context before the first handshake. When true, the handshake suspends once at ClientHello. Default false (no behavior change).

2. TlsOperationStatus.NeedsClientHello

namespace System.Net.Security;

public enum TlsOperationStatus
{
        Complete = 0,
        WantRead = 1,
        WantWrite = 2,
        Closed = 3,
        WantCredentials = 4,
        NeedsCertificateValidation = 5,
        NeedsServerOptions = 6,

        /// <summary>
        /// Server-side only. The handshake paused at ClientHello because raw ClientHello
        /// inspection was enabled on the <see cref="TlsContext"/>. Retrieve the raw bytes
        /// via <see cref="TlsSession.GetClientHelloBytes"/>, then call
        /// <see cref="TlsSession.Handshake"/> (fd mode) or
        /// <see cref="TlsSession.ProcessHandshake"/> again to resume the handshake.
        /// </summary>
+        NeedsClientHello = 7,
}

New suspension reason returned by Handshake() / ProcessHandshake(). Means: the ClientHello has arrived and is available; call GetClientHelloBytes(), then call Handshake() / ProcessHandshake() again to resume.

3. TlsSession.GetClientHelloBytes()

namespace System.Net.Security;

public sealed partial class TlsSession
{
+ public ReadOnlySpan<byte> GetClientHelloBytes();
}

Returns the raw ClientHello handshake message (handshake type + 3-byte length + body; no 5-byte TLS record header). Zero-copy view over OpenSSL's own buffer; valid until the next Handshake() / ProcessHandshake() call. Empty if inspection wasn't enabled or the ClientHello hasn't arrived yet.

---

Under the hood

When EnableClientHelloInspection is set, the server's SSL_CTX is armed with two OpenSSL callbacks: SSL_CTX_set_msg_callback (copies the inbound ClientHello bytes into per-SSL ex_data) and SSL_CTX_set_client_hello_cb (returns SSL_CLIENT_HELLO_RETRY exactly once). That makes SSL_do_handshake return SSL_ERROR_WANT_CLIENT_HELLO_CB, which the managed layer maps to TlsOperationStatus.NeedsClientHello. GetClientHelloBytes() just returns a span over the already-captured ex_data buffer — no parsing, no copy. The next Handshake() / ProcessHandshake() re-enters the suspended SSL_do_handshake, the callback now returns "success," and the handshake proceeds normally (emits ServerHello, etc.).

How it maps to the two drive modes

• fd-mode (Handshake(), OpenSSL owns the socket): suspends after OpenSSL reads the ClientHello off the fd itself.
• BIO-mode (ProcessHandshake(input, output, ...), caller owns the socket): the ClientHello is consumed from the input span into OpenSSL's BIO, then it suspends. On resume the caller passes empty input (the bytes were already consumed) and OpenSSL writes the ServerHello to the output span.

Typical caller loop (fd-mode)

using TlsContext ctx = TlsContext.Create(serverOptions);
ctx.EnableClientHelloInspection = true;                 // (1) opt in
using TlsSession server = TlsSession.Create(ctx, socketHandle);

while (true)
{
    TlsOperationStatus status = server.Handshake();
    switch (status)
    {
        case TlsOperationStatus.NeedsClientHello:        // (2) suspended once
            ReadOnlySpan<byte> hello = server.GetClientHelloBytes();
            InspectSniAlpn(hello);                       //     Kestrel parses it itself
            continue;                                    //     loop -> resumes
        case TlsOperationStatus.Complete:
            goto done;
        // ...WantRead / WantWrite / NeedsCertificateValidation...
    }
}
done: ;