Establish connection with HttpClient without making call. Also track status of connection

[JamesNK]

Background and Motivation

An important gRPC feature in .NET 6 is adding client side load balancing. Issue discussing why is here - dotnet/core#5495.

tldr: Kubernetes pods can scale to multiple instances. Kubernetes built in load balancer is L4. That means it doesn't distribute load well with HTTP/2 because of multiplexing.

Part of load balancing is discovering whether a connection can be successfully established with the server. For example, a client might be configured with 3 endpoints, and it will use the first endpoint it can successfully connect to.

The gRPC spec for connectivity status is here: https://github.com/grpc/grpc/blob/master/doc/connectivity-semantics-and-api.md

Proposed API

Add a new method to HttpClient that can be used to establish a connection to an endpoint without making a HTTP request.

Also the returned HttpConnection provides the ability to track the state of the connection to the server.

public class SocketsHttpHandler
{
    public Task<HttpConnection> ConnectAsync(Uri, CancellationToken);
}

public class HttpConnection
{
    public Uri ServerUri { get; }
    public ConnectionState State { get; }
    
    // Plus an API that allows you to subscribe to state change updates.
    // I copied this off https://docs.microsoft.com/en-us/dotnet/api/system.threading.cancellationtoken.register
    public IDisposable StateChanged(Action action);
    public IDisposable StateChanged(Action<object> action, Object state);
}

public enum ConnectionState
{
    // ...
}

The returned connection would represent the HTTP connection to the server. So for HTTP/2 and HTTP/3 a successfully connected status would include exchanging SETTINGS frames.

Things I'm not sure about:

• Is the returned connection an abstraction over multiple internal connections to the server? For example, there could be multiple HTTP/2 connections to http://localhost. If any have a state of Connected then the abstraction has a state of Connected?
• Similar to question above, how would this feature work in HTTP/1.1 (TCP connection per call) and HTTP/3 (UDP world)
• You have a connection that is no longer connected. How to reconnect? Do you call ConnectAsync again with the same Uri? Does it return the same HttpConnection instance?
• Should ConnectAsync be async and start a connection? Or should it sync and return a HttpConnection that then has a ConnectAsync method on it?

Usage Examples

Basic usage (this would allow GrpcChannel to support a ConnectAsync method):

var client = new HttpClient();
var connection = await client.ConnectAsync("https://localhost");
if (connection.State != ConnectionState.Connected)
{
    throw new InvalidOperationException("Could not connect to server.");
}

// Request made using previous established connection (open connection fetched from pool using current behavior)
var response = await client.GetAsync("https://localhost/settings.json");

Basic load balancing that uses the first successful connection (this is called a pick first strategy):

var client = new HttpClient();
var endpoints = new string[] { "https://localhost", "https://localhost:5000", "https://localhost:5001" };

foreach (var endpoint in endpoints)
{
    var connection = await client.ConnectAsync(endpoint);
    if (connection.State == ConnectionState.Connected)
    {
        return await client.GetAsync(endpoint + "/settings.json");
    }
}

throw new InvalidOperationException("Could not connect to the configured servers");

Risks

In the future we might want to implement channelz. channelz is about collecting gRPC call stats. You can view calls made down to a subchannel level (subchannel = individual TCP connection to a server). There hasn't been much demand for it (one issue with no upvotes) so right now it doesn't seem important.

Should consider a design that doesn't block channelz in the future.

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Background and Motivation

An important gRPC feature in .NET 6 is adding client side load balancing. Issue discussing why is here - dotnet/core#5495.

tldr: Kubernetes pods can scale to multiple instances. Kubernetes built in load balancer is L4. That means it doesn't distribute load well with HTTP/2 because of multiplexing.

Part of load balancing is discovering whether a connection can be successfully established with the server. For example, a client might be configured with 3 endpoints, and it will use the first endpoint it can successfully connect to.

Proposed API

Add a new method to HttpClient that can be used to establish a connection to an endpoint with making a HTTP request.

Also the returned HttpConnection provides the ability to track the state of the connection to the server.

public class HttpClient
{
    public Task<HttpConnection> HttpClient.ConnectAsync(Uri, CancellationToken);
}

public class HttpConnection
{
    public Uri ServerUri { get; }
    public ConnectionState State { get; }
    
    // Plus an API that allows you to subscribe to state change updates.
    // I copied this off https://docs.microsoft.com/en-us/dotnet/api/system.threading.cancellationtoken.register
    public IDisposable StateChanged(Action action);
    public IDisposable StateChanged(Action<object> action, Object state);
}

public enum ConnectionState
{
    // ...
}

The returned connection would represent the HTTP connection to the server. So for HTTP/2 and HTTP/3 a successfully connected status would include exchanging SETTINGS frames.

Things I'm not sure about:

• Is the returned connection an abstraction over multiple internal connections to the server? For example, there could be multiple HTTP/2 connections to http://localhost. If any have a state of Connected then the abstraction has a state of Connected?
• Similar to question above, how would this feature work in HTTP/1.1 (TCP connection per call) and HTTP/3 (UDP world)
• You have a connection that is no longer connected. How to reconnect? Do you call ConnectAsync again with the same Uri? Does it return the same HttpConnection instance?
• Should ConnectAsync be async and start a connection? Or should it sync and return a HttpConnection that then has a ConnectAsync method on it?

Usage Examples

Basic usage (this would allow GrpcChannel to support a ConnectAsync method):

var client = new HttpClient();
var connection = await client.ConnectAsync("https://localhost");
if (connection.State != ConnectionState.Connected)
{
    throw new InvalidOperationException("Could not connect to server.");
}

// Request made using previous established connection
var response = await client.GetAsync("https://localhost/settings.json");

Basic load balancing that uses the first successful connection (this is called a pick first strategy):

var client = new HttpClient();
var endpoints = new string[] { "https://localhost", "https://localhost:5000", "https://localhost:5001" };

foreach (var endpoint in endpoints)
{
    var connection = await client.ConnectAsync(endpoint);
    if (connection.State == ConnectionState.Connected)
    {
        return await client.GetAsync(endpoint + "/settings.json");
    }
}

throw new InvalidOperationException("Could not connect to the configured servers");

Risks

In the future we might want to implement channelz. channelz is about collecting gRPC call stats. You can view calls made down to a subchannel level (subchannel = individual TCP connection to a server). There hasn't been much demand for it (one issue with no upvotes) so right now it doesn't seem important.

Should consider a design that doesn't block channelz in the future.

Author:	JamesNK
Assignees:	-
Labels:	api-suggestion, area-System.Net.Http
Milestone:	-

[scalablecory]

I experimented with something very similar for YARP a while back and landed on an API that can provide some food for thought. It works by separating reservation of a request from sending the request. You would use it something like this:

HttpClient client = ...;
HttpRequestMessage request = ...;
HttpResponseMessage response;

if(client.TryReserve(request, out HttpRequestTicket? requestTicket))
{
    // a connection was open, HTTP/2 stream was available, etc.
    // the connection might still get closed by the time you call SendAsync, though, so prepare to retry...
    response = await requestTicket.SendAsync();
}
else
{
    // otherwise, just send it out.
   response = await client.SendAsync(request);
}

This way you could test multiple IPs for an available connection:

HttpClient client = ...;
HttpRequestMessage request = ...;
Uri[] uris = ...;
Random rng = ...;
HttpRequestTicket? requestTicket = null;

for(int i = 0; i < uris.Length; ++i)
{
    request.RequestUri = uris[i];
    if(client.TryReserve(request, out requestTicket))
        break;
}

HttpResponseMessage response;
if(requestTicket is not null)
{
    response = await requestTicket.SendAsync();
}
else
{
   request.RequestUri = uris[rng.Next(uris.Length)];
   response = await client.SendAsync(request);
}

This combined with something like:

class SocketsHttpHandler
{
     public Action<ConnectionPoolKey, DisconnectReason> DisconnectingCallback { get; set; }
     public Task ConnectAsync(ConnectionPoolKey key);

     public Action<ConnectionPoolKey> ConnectedCallback { get; set; } // ?
     public Task DisconnectAsync(ConnectionPoolKey key); // ?

     public Action<ConnectionPoolKey, ConnectionState> ConnectionStateChangedCallback { get; set; } // ?
}

enum DisconnectReason
{
     IdleTimeout, ServerGoAway, ConnectionReset,
     // ...
}

I think would get us very far -- I think all the functionality James is requesting here -- without limiting/exposing too much our internal connection pooling designs.

[scalablecory]

I've also been prototyping a lower-level, higher-perf HttpConnection API that could accomplish what you're doing, though HTTP/2 support hasn't been added yet: https://github.com/scalablecory/NetworkToolkit/

[geoffkizer]

Is the returned connection an abstraction over multiple internal connections to the server? For example, there could be multiple HTTP/2 connections to http://localhost. If any have a state of Connected then the abstraction has a state of Connected?

I assume the answer here is yes -- that seems to match what the gRPC connectivity spec expects.

That means this is actually more like "connection pool status" as opposed to connection status.

[JamesNK]

I've been thinking about what the minimum viable product could be to support gRPC load balancing with SocketsHttpHandler.

I think the requirements could be simplified down to one method to ensure there is a connection, and combine it with using SocketsHttpHandler.ConnectCallback to track connection status.

public class SocketsHttpHandler
{
    public Task EnsureConnectionAsync(Uri, CancellationToken);
}

EnsureConnectionAsync will either establish a connection to the target URI if one doesn't exist, or return immediately if one exists in the connection pool. When establishing a connection it will do all the standard behavior to create connection that would otherwise happen when making a request. The only difference is no request is made once at the end. A completed task returns once the TCP connection was made, TLS negotated, HTTP/2 or HTTP/3 connection negotated, etc.

Connection status tracking would then work by using SocketsHttpHandler.ConnectCallback. The callback returns a stream. I can wrap the stream in one of my own and store it somewhere to track whether it has been disposed.

To do this I would need to reimplement the underlying logic of ConnectCallback - for HTTP/1.1 and HTTP/2 this is just creating a TCP socket. I haven't looked into what would be needed for creating a HTTP/3 connection.

Connection health from a gRPC load balancer point of view:

• If there are no stored streams for a URI then there is no connection to it.
• Connection status is healthy once ConnectCallback has been called, wrapped stream has been stashed and EnsureConnectionAsync has returned successfully. Need both to happen because the callback only verifies the transport is ok. Once EnsureConnectionAsync has returned then we know TLS, HTTP version, and HTTP connection have been negotiated successfully.
• Connection status reverts to disconnected if all the wrapped streams for a URI are disposed by SocketsHttpHandler. Multiple streams could be created for HTTP/2 under high-load but it is simple enough to track the status of all of them and aggregate that status.

What do you think? @scalablecory @geoffkizer