SignalR how to reconnect with same connectionId

[quytrieu37]

Hi everyone, my question is related to signalR.
I have a long-running task on Hub and after it is complete, the hub will send a response base on Clients.Caller.SendAsync method
But currently, while running, it look like that hub doesn't send keep-alive message to client and after ServerTimeout (default is 30s), the connection will timeout and reconnect cause I config WithAutomaticReconnect() on client.
But when after reconnecting, the connection comes with difference connectionId so it can't listen to messages from server which send to old connectionId
I understand that there are Groups and Users to handle this scenario, but perhaps there is an alternative approach to retaining the connection ID after reconnecting?
Thank you!

[davidfowl]

https://learn.microsoft.com/en-us/aspnet/core/release-notes/aspnetcore-8.0?view=aspnetcore-8.0#signalr-stateful-reconnect

[achalrajpoot1]

@davidfowl I shared the details as above . Could you please have a look ?

[davidfowl]

cc @BrennanConroy

[achalrajpoot1]

@davidfowl @BrennanConroy I am blocked as of now for the reconnect implementation. I would request you please have a look on this issue. Thank you .

[BrennanConroy]

What are you blocked by? Stateful reconnect is not a guarantee that clients will reconnect with the same ID. It is a best effort attempt (and that effort may change/improve in future versions) and if your application is relying on the server maintaining state for clients across reconnects then you need to implement that logic yourself.

[Mise22]

I also can't get stateful reconnect to work, is there a code sample with the js client of this working?

In NegotiationResponse.cs it is suggested that the client should handle connectionId preservation:
"UseStatefulReconnect -> If set, the connection should attempt to reconnect with the same ConnectionId if it disconnects."

Looking at the client source (HubConnection.ts/HttpConnection.ts) I see no code preserving the connection id (this.connectionId = undefined; is called before reconnect), is this done in a different way?

I have enabled withAutomaticReconnect() because without it there is no reconnection at all (also looking at the code). In the docs it is unclear if it should be enabled to use withStatefulReconnect(), it looked to me like it is a parallel feature https://learn.microsoft.com/en-us/aspnet/core/release-notes/aspnetcore-8.0?view=aspnetcore-8.0#signalr-stateful-reconnect

[co-dax]

As a workaround you should be able to use IUserIdProvider and send the message to the specific client by using the "SignalR UserId" you returned from IUserIdProvider.GetUserId(HubConnectionContext connection). That way you would send the message to the client by using the SignalR UserId that has been generated by you (so you always know its value) instead of having to rely on SignalR connectionId. If you use IUserIdProvider.GetUserId(HubConnectionContext connection) then SignalR will automatically and internally link the SignalR UserId you returned from it to the SignalR connectionId.

[ghost]

Hi @BrennanConroy, I have also been unable to get stateful re-connect to work as far as I understand it should work.

I have setup a sample here

The steps I am using to work with the sample are as follows:

1. Start the solution which should fire up the aspnet core server and angular client
2. When the client loads you should start seeing a counter incrementing, this counter is being pushed to the client via a hosted service which sends messages to a group using the hub context.
3. I disable networking using the dev tools and wait for the signalr connection to close.
4. When I enable networking again I am expecting any messages which I did not receive while in the disconnected state to be received, which does not seem to be the case. You can see in the screen shot below that I did not receive a number of counter messages once the connection re-established.

Please can you confirm if there is something which I have done incorrectly, or if my expectations of how this feature works is incorrect.

Thanks in advance!

[ITS-GarethB]

I am seeing the same experience as @jonesr-out . Either stateful reconnect is broken or my understanding of what it does is wrong.

[EnCey]

Hi @BrennanConroy,

like the posters before me, I cannot get stateful reconnect to work: the ConnectionId is not maintained. We use the ConnectionId to associate state with the connection and send messages to specific clients (is there any other way?).

The automatic reconnect kicks in after the serverTimeout is exceeded, but the response always contains a different ConnectionId.

This is simple to verify by configuring the client's withServerTimeout to be lower than the server's KeepAliveInterval, sending it into an infinite loop of re-negotiating and re-connecting, each time with a different ConnectionId:

On the server side, a simple Debug.WriteLine(Context.ConnectionId) reveals that OnDisconnected is called each time, followed by an OnConnected with a new ID.

(This wasn't the only way I tested, also happens with regular timeouts and throttling in DevTools to block server KeepAlive pings until the serverTimeout is hit)

---

If maintaining the Connection ID is not a goal of stateful reconnect, then what is the purpose of the feature?
Should we identify and talk to clients in a different way?
How does recognizing when a connection is up and replaying messages work when the connection is treated as new and different – or is there some link to the "previous" connection that we could evaluate?

[EnCey]

FWIW, I did some digging around in the source and found confirmation that re-using the ConenctionId was at least intended:

in NegotiationResponse:

    /// <summary>
*    /// If set, the connection should attempt to reconnect with the same <see cref="BaseConnectionContext.ConnectionId"/> if it disconnects.
    /// It should also set <see cref="IStatefulReconnectFeature"/> on the <see cref="BaseConnectionContext.Features"/> collection so other layers of the
    /// application (like SignalR) can react.
    /// </summary>
    public bool UseStatefulReconnect { get; set; }

and HttpConnectionOptions

    /// <summary>
    /// Setting to enable Stateful Reconnect between client and server, this allows reconnecting that preserves messages sent while disconnected.
*    /// Also preserves the <see cref="HttpConnection.ConnectionId"/> when the reconnect is successful.
    /// </summary>
    /// <remarks>
    /// Only works with WebSockets transport currently.
    /// </remarks>
    public bool UseStatefulReconnect { get; set; }

However in HttpConnectionManager a new ID is always generated (and a new connectionToken too)
(Also note that this does receive a useStatefulReconnect parameter, but doesn't use it)

   internal HttpConnectionContext CreateConnection(HttpConnectionDispatcherOptions options, int negotiateVersion = 0, bool useStatefulReconnect = false)
    {
        string connectionToken;
        var id = MakeNewConnectionId();
        if (negotiateVersion > 0)
        {
            connectionToken = MakeNewConnectionId();
        }
        else
        {
            connectionToken = id;
        }

        var metricsContext = _metrics.CreateContext();

        Log.CreatedNewConnection(_logger, id);

        var pair = CreateConnectionPair(options.TransportPipeOptions, options.AppPipeOptions);
        var connection = new HttpConnectionContext(id, connectionToken, _connectionLogger, metricsContext, pair.Application, pair.Transport, options, useStatefulReconnect);

        _connections.TryAdd(connectionToken, connection);

        return connection;
    }

The HttpConnectionDispatcher calls this method; it does have a GetConnectionToken method and uses it in one instance to obtain an already existing connection:

private static StringValues GetConnectionToken(HttpContext context) => context.Request.Query["id"];

// [...]

    // This is only used for WebSockets connections, which can connect directly without negotiating
    private async Task<HttpConnectionContext?> GetOrCreateConnectionAsync(HttpContext context, HttpConnectionDispatcherOptions options)
    {
        var connectionToken = GetConnectionToken(context);
        HttpConnectionContext? connection;

        // There's no connection id so this is a brand new connection
        if (StringValues.IsNullOrEmpty(connectionToken))
        {
            connection = CreateConnection(options);
        }
        // Use ToString; IsNullOrEmpty doesn't tell the compiler anything about implicit conversion to string.
        else if (!_manager.TryGetConnection(connectionToken.ToString(), out connection))
        {
            // No connection with that ID: Not Found
            context.Response.StatusCode = StatusCodes.Status404NotFound;
            await context.Response.WriteAsync("No Connection with that ID");
            return null;
        }

        return connection;
    }

I have no experience with the whole code base, but it seems to me that all the pieces may be there and the feature just hasn't been fully implemented?

In the TS client, resolveNegotiateUrl appends the useStatefulReconnect parameter and is called indirectly (via _getNegotiationResponse) in _createTransport which has a negotiationResponse from which it derives its own connectionId.

My point / question is, could not the id= parameter be added to the negotiate request so that the HttpConnectionDispatcher can use it to check if a connection with this ID exists and if so, re-use it?

[ITS-GarethB]

From my limited testing, this feature doesn't work as I would have expected either but...

With stateful reconnect off, if client sends a message while disconnected temporarily, the message is lost and not received by the recipients.
With it on, the messages are buffered until the client reconnects and then sent. If the ClientTimeoutInterval expires and the client still hasn't reconnected, the messages are lost.

Thats all it does, it does not guarantee connection Id is persisted on reconnect. On reconnect is only hit if the ClientTimeoutInterval expires which shouldn't be the norm but more edge cases anyway and increasing the ClientTimeoutInterval might be enough to solve at least some reconnects from occuring. Persisting connection ID is something we've had to handle ourselves.

[davidfowl]

Yes that’s correct. If you need reliable messaging then you need durable storage. SignalR is a transport abstraction, not a durable messaging system. Your application can persist whatever messages it wants to replay indefinitely.

[EnCey]

Our use case is somewhat different: we have expensive, client-specific state on the server that we wish to re-use when a client reconnects. We can recreate that state (necessary when client disconnects for longer period), but that is suboptimal when the client only has a brief network hiccup and reconnects immediately – hence our interest in the ConnectionId to reclaim that state.
(we aren't that interested in the message buffer aspect, that's a neat bonus but not our primary concern)

I don't understand yet how messages are buffered and resent if the server doesn't know who it is talking to?
What ID or mechanism is used to associate a reconnected client with the message buffer? Or are messages only buffered and resent from the client-side?

Personally I find the feature description and also the XML comments of UseStatefulReconnect quite misleading, as they explicitly say that ConnectionId is re-used. Is there any interest in implementing that? If not, I would at least suggest to update the comments to clarify the behavior

[BrennanConroy]

I stated this above, but I'll restate it. Stateful reconnect is not a guarantee that clients will reconnect with the same ID. It is a best effort attempt and will fallback to automatic reconnect if enabled or just close the connection and let client code handle the closure. Best effort can definitely be improved in the future, but should never be relied on.

The main way I test this locally is to run a proxy like Fiddler and abort the websocket connection from the UI.

Purposefully setting withServerTimeout to a low value does not create the conditions for stateful reconnect to occur at this time. This may change in the future.

You can see from the devtools that the connection token is preserved which results in the server using the same connection ID.

[EnCey]

Thank you for the clarification Brennan. It was clear to me that there can't be a guarantee that the same ConnectionID is always re-used, but I wrongfully assumed that it was never re-used, as I could not reproduce the scenario in which it works.

Besides withServerTimeout I tried closing the underlying WS connection with connection.transport._webSocket.close(), thinking this would simulate a low-level disconnect, but apparently not so.

It would be neat if stateful reconnect worked in more cases, as it would simplify our session management. We have to handle "hard" disconnects and timeouts anyway, but short-lived, temporary disconnects could have been delegated to SignalR if the ConnectionID would be re-used more. In our case, we want to give clients a chance to reconnect to the same session until a timeout is hit. For now, we will solve this in client code by introducing our own session ID and mapping connection IDs to it.