Skip to content

GRPC Service (Server Side) not causing exception if unable to send data to client #334

Open
@Uight

Description

@Uight

Im having a problem with server to client communication. I have a Server implementing the following interface method:

[Service]
public interface ICommandApiService
{
IAsyncEnumerable SubscribeForUserDialogEvents(CallContext context = default);
}

    public async IAsyncEnumerable<UserDialogEventData> SubscribeForUserDialogEvents(CallContext context = default)
    {
        var cancel = context.CancellationToken;
        var clientSubscription = new ClientUserDialogSubscription();
        clientSubscriptions.Add(clientSubscription);
        
        logger.Info($"A subscriber for user dialog events connected. Current subscriber count {clientSubscriptions.Count}");
        var delayCancellationTokenSource = CancellationTokenSource.CreateLinkedTokenSource(cancel);
        var waitForCancellationTask = Task.Delay(Timeout.Infinite, delayCancellationTokenSource.Token);

        try
        {
            // Immediately push a success message so the client can verify the connection is up and running.
            yield return new UserDialogEventData
            {
                EventType = UserDialogEventType.ConfirmSuccess
            };

            //Push messages for all currently opened dialogs
            foreach (var openDialog in currentlyOpenedDialogs)
            {
                yield return openDialog.Value;
            }

            var nextTcs = clientSubscription.UserDialogEventRequestBatch.NextBatch;

            while (!cancel.IsCancellationRequested)
            {
                var notificationTask = nextTcs.Task;
                var heartbeatTaskDelay = Task.Delay(2500, cancel);
                var completedTask = await Task.WhenAny(notificationTask, heartbeatTaskDelay, waitForCancellationTask);

                if (cancel.IsCancellationRequested)
                {
                    break;
                }

                if (completedTask == heartbeatTaskDelay)
                {
                    yield return new UserDialogEventData
                    {
                        EventType = UserDialogEventType.KeepAlive
                    };
                    continue;
                }

                var notificationBatch = await nextTcs.Task;
                nextTcs = notificationBatch.NextBatch;

                //Send new event to client
                yield return notificationBatch.UserDialogEventRequest;

                // Notify successfully send
                clientSubscription.AcknowledgmentTcs.SetResult(true);
            }
        }
        finally
        {
            clientSubscriptions.Remove(clientSubscription);
            clientSubscription.AcknowledgmentTcs.SetResult(false);
            logger.Info("A subscriber for user dialog events disconnected");
            
            await delayCancellationTokenSource.CancelAsync();
            waitForCancellationTask.Wait(TimeSpan.FromMilliseconds(100));
            delayCancellationTokenSource.Dispose();
        }
    }

this works fine for the most part and also when the client cleanly disconnects this works fine. This sends out new Data when triggered from another method and stops when the cancellation is requested. But we identified one problem here which is why we added the Keepalive messages all 2,5 seconds but that doesnt help.

The problem is that in the production environment our software runs in sometimes the connection is lost. (Most easily reproduced by just disconnecting the LAN Cable between client and server. In this case i would expect that in end up failing at on of the yield returns. However the yield returns just keep working and the loop things its still sending our data even after 10 Minutes it doesn't fail.
The client side pretty much immediately recognizes the connection loss but the server never does.
The most problematic for us is that we tried to monitor the send options to check if every connected client did receive the message. But in case of a connection loss we still think we can send out without an error but that isn't the case.

Obviosly there would be some Solutions here like using a two way stream with acknowledgement and timeout or similar things. But i would still like to know why the server doesnt recognize the connection loss. Is the code wrong? or is the server setup badly?

Server setup:

        var builder = WebApplication.CreateBuilder();
        var services = builder.Services;
        
        services.AddSingleton(typeof(CommandApiService), commandService);
        services.AddGrpc(options =>
        {
            const int fiftyMegaBytes = 50 * 1024 * 1024;
            options.MaxReceiveMessageSize = fiftyMegaBytes;
            options.MaxSendMessageSize = fiftyMegaBytes;
            options.EnableDetailedErrors = true;
        });
        
        services.AddCodeFirstGrpc(config => { config.ResponseCompressionLevel = System.IO.Compression.CompressionLevel.Optimal; });
        
        builder.WebHost.ConfigureKestrel(options =>
        {
            options.ListenAnyIP(port, listenOptions => listenOptions.Protocols = HttpProtocols.Http2);
            options.Limits.MaxRequestBodySize = 50 * 1024 * 1024;
        });
        
        app = builder.Build();
        app.UseRouting();
        
        app.MapGrpcService<CommandApiService>(); 
        
        app.RunAsync();

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions