Stream reader MoveNext() doesn't comply with gRPC spec, throws exception on underlying TCP/HTTP2 stream abort/close

[markbearden]

Using Grpc.AspNetCore v2.33.1 (in .NET Core 3.1 project).

The behavior of async stream reading for an open gRPC connection doesn't match the main gRPC API documentation in the following respect. The API doc for IAsyncStreamReader says (with my bold emphasis):

On the server side, MoveNext() does not throw exceptions. In case of a failure, the request stream will appear to be finished (MoveNext will return false) and the CancellationToken associated with the call will be cancelled to signal the failure.

But the .NET implementation of HttpContextStreamReader.MoveNext() does throw an exception if the underlying TCP/HTTP2 transports are abnormally closed.

I am experiencing this, and I found another case that looks like mine already posted publicly here, so I'll refer to that in lieu of posting my own error detail. (Search for the System.IO.Exception error returned from the "FullDuplexCall" unit test, and note how HttpContextStreamReader`1.<MoveNext> is in the exception stack trace. A snip of the exception message is copied below.)

I can clearly see the exception-throwing in the current source as follows:

• Within MoveNext(), this call to ReadStreamMessageAsync() is not enclosed with a try/catch.
• Within ReadStreamMessageAsync(), any exception that's not an OperationCanceledException is rethrown within the catch.

Failure leading to abrupt closure of the underlying transport (in .NET Core at least) produces a System.IO.Exception that is re-thrown and escapes the MoveNext() call, contrary to the API doc's promise that MoveNext() would return false "in case of a failure".

Additional detail about the context in which I'm seeing MoveNext() throw System.IO.Exception, if it matters: In my project, I'm calling MoveNext() on the same Grpc.AspNetCore.Server.Internal.HttpContextStreamReader instance that is passed into my override of Grpc.Core.Interceptors.Interceptor.DuplexStreamingServerHandler. In .NET Core environment using Microsoft.Extensions.Hosting and Microsoft.AspNetCore.Server.Kestrel.Core to initialize a server-side gRPC node that is accepting a duplex-streaming service call (and then the client abruptly closes the underlying TCP connection).

---

Also here I preserve a snip of the public error case referenced earlier's exception trace:
fail: Grpc.Testing.TestServiceImpl[3] Error when executing service method 'FullDuplexCall'. System.IO.IOException: The client reset the request stream. at System.IO.Pipelines.PipeCompletion.ThrowLatchedException() at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result) at System.IO.Pipelines.Pipe.GetReadAsyncResult() at System.IO.Pipelines.Pipe.DefaultPipeReader.GetResult(Int16 token) at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2MessageBody.ReadAsync(CancellationToken cancellationToken) at Grpc.AspNetCore.Server.Internal.PipeExtensions.ReadStreamMessageAsync(PipeReader input, HttpContextServerCallContext context, CancellationToken cancellationToken) in /var/local/git/grpc-dotnet/src/Grpc.AspNetCore.Server/Internal/PipeExtensions.cs:line 271 at System.Threading.Tasks.ValueTask`1.get_Result() at Grpc.AspNetCore.Server.Internal.HttpContextStreamReader`1.<MoveNext>g__MoveNextAsync|10_0(ValueTask`1 readStreamTask) in /var/local/git/grpc-dotnet/src/Grpc.AspNetCore.Server/Internal/HttpContextStreamReader.cs:line 48

[BrennanConroy]

Triage: We have a pattern where we allow configuration of the behavior similar to this. We might need to apply the same pattern here.

[JunTaoLuo]

I was thinking of ThrowOperationCanceledOnCancellation when I mentioned this during triage where we configure between different Exception types due to discrepancies between gRPC and .NET conventions. However, here we might need to just modify the behaviour to be spec compliant.

[halter73]

How do the gRPC and .NET conventions differ? I think we should throw OperationCanceledExceptions less than we currently do in Kestrel. See dotnet/aspnetcore#4747 (comment)