INTERNAL: Missing gRPC status code

[novoj]

When I upgrade Armeria server from 1.32.3 to 1.32.4 I start to get following error on the client on particular gRPC call:

Caused by: io.grpc.StatusRuntimeException: INTERNAL: Missing gRPC status code
	at io.grpc.Status.asRuntimeException(Status.java:532)
	at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:564)
	at com.linecorp.armeria.internal.client.grpc.ArmeriaClientCall.lambda$close$14(ArmeriaClientCall.java:556)
	at com.linecorp.armeria.internal.shaded.guava.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
	at com.linecorp.armeria.internal.client.grpc.ArmeriaClientCall.close(ArmeriaClientCall.java:553)
	at com.linecorp.armeria.internal.client.grpc.ArmeriaClientCall.closeWhenEos(ArmeriaClientCall.java:513)
	at com.linecorp.armeria.internal.client.grpc.ArmeriaClientCall.lambda$transportReportStatus$11(ArmeriaClientCall.java:469)
	at com.linecorp.armeria.internal.shaded.guava.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
	at com.linecorp.armeria.internal.client.grpc.ArmeriaClientCall.transportReportStatus(ArmeriaClientCall.java:469)
	at com.linecorp.armeria.internal.common.grpc.TransportStatusListener.transportReportStatus(TransportStatusListener.java:34)
	at com.linecorp.armeria.internal.common.grpc.HttpStreamDeframer.processOnComplete(HttpStreamDeframer.java:159)
	at com.linecorp.armeria.internal.common.stream.DecodedStreamMessage$DecodingSubscriber.onComplete(DecodedStreamMessage.java:288)
	at com.linecorp.armeria.common.stream.DeferredStreamMessage$ForwardingSubscriber.onComplete(DeferredStreamMessage.java:526)
	at com.linecorp.armeria.internal.common.stream.SurroundingPublisher$SurroundingSubscriber.close0(SurroundingPublisher.java:447)
	at com.linecorp.armeria.internal.common.stream.SurroundingPublisher$SurroundingSubscriber.finalize(SurroundingPublisher.java:323)
	at com.linecorp.armeria.internal.common.stream.SurroundingPublisher$SurroundingSubscriber.sendTail(SurroundingPublisher.java:304)
	at com.linecorp.armeria.internal.common.stream.SurroundingPublisher$SurroundingSubscriber.publish(SurroundingPublisher.java:288)
	at com.linecorp.armeria.internal.common.stream.SurroundingPublisher$SurroundingSubscriber.onComplete(SurroundingPublisher.java:399)
	at com.linecorp.armeria.common.DefaultContextAwareRunnable.run(DefaultContextAwareRunnable.java:45)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:166)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
	at --- Async.Stack.Trace --- (captured by IntelliJ IDEA debugger)
	at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java)
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:836)
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute0(SingleThreadEventExecutor.java:827)
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:817)
	at com.linecorp.armeria.common.AbstractContextAwareExecutor.execute(AbstractContextAwareExecutor.java:81)
	at com.linecorp.armeria.internal.common.SplitHttpMessageSubscriber.onComplete(SplitHttpMessageSubscriber.java:256)
	at com.linecorp.armeria.common.stream.CancellableStreamMessage$CloseEvent.notifySubscriber(CancellableStreamMessage.java:262)
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriberOfCloseEvent0(DefaultStreamMessage.java:311)
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriberOfCloseEvent(DefaultStreamMessage.java:302)
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.handleCloseEvent(DefaultStreamMessage.java:446)
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriber0(DefaultStreamMessage.java:389)
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.doRequest(DefaultStreamMessage.java:285)
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.request(DefaultStreamMessage.java:267)
	at com.linecorp.armeria.common.stream.CancellableStreamMessage$SubscriptionImpl.request(CancellableStreamMessage.java:215)
	at com.linecorp.armeria.internal.common.SplitHttpMessageSubscriber.request0(SplitHttpMessageSubscriber.java:194)
	at com.linecorp.armeria.internal.common.SplitHttpMessageSubscriber.request(SplitHttpMessageSubscriber.java:183)
	at com.linecorp.armeria.internal.common.stream.SurroundingPublisher$SurroundingSubscriber.requestUpstream(SurroundingPublisher.java:339)
	at com.linecorp.armeria.internal.common.stream.SurroundingPublisher$SurroundingSubscriber.publish(SurroundingPublisher.java:283)
	at com.linecorp.armeria.internal.common.stream.SurroundingPublisher$SurroundingSubscriber.publishDownstream(SurroundingPublisher.java:367)
	at com.linecorp.armeria.internal.common.stream.SurroundingPublisher$SurroundingSubscriber.onNext(SurroundingPublisher.java:384)
	at com.linecorp.armeria.internal.common.SplitHttpMessageSubscriber.onNext0(SplitHttpMessageSubscriber.java:239)
	at com.linecorp.armeria.internal.common.SplitHttpMessageSubscriber.lambda$onNext$2(SplitHttpMessageSubscriber.java:228)
	at com.linecorp.armeria.common.DefaultContextAwareRunnable.run(DefaultContextAwareRunnable.java:45)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:166)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
	at --- Async.Stack.Trace --- (captured by IntelliJ IDEA debugger)
	at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java)
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:836)
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute0(SingleThreadEventExecutor.java:827)
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:817)
	at com.linecorp.armeria.common.AbstractContextAwareExecutor.execute(AbstractContextAwareExecutor.java:81)
	at com.linecorp.armeria.internal.common.SplitHttpMessageSubscriber.onNext(SplitHttpMessageSubscriber.java:228)
	at com.linecorp.armeria.internal.common.DefaultSplitHttpResponse$SplitHttpResponseBodySubscriber.onNext(DefaultSplitHttpResponse.java:98)
	at com.linecorp.armeria.internal.common.DefaultSplitHttpResponse$SplitHttpResponseBodySubscriber.onNext(DefaultSplitHttpResponse.java:70)
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriberWithElements(DefaultStreamMessage.java:420)
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriber0(DefaultStreamMessage.java:398)
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.doRequest(DefaultStreamMessage.java:285)
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.request(DefaultStreamMessage.java:267)
	at com.linecorp.armeria.common.stream.CancellableStreamMessage$SubscriptionImpl.request(CancellableStreamMessage.java:215)
	at com.linecorp.armeria.internal.common.SplitHttpMessageSubscriber.request0(SplitHttpMessageSubscriber.java:194)
	at com.linecorp.armeria.internal.common.SplitHttpMessageSubscriber.request(SplitHttpMessageSubscriber.java:183)
	at com.linecorp.armeria.internal.common.stream.SurroundingPublisher$SurroundingSubscriber.requestUpstream(SurroundingPublisher.java:339)
	at com.linecorp.armeria.internal.common.stream.SurroundingPublisher$SurroundingSubscriber.publish(SurroundingPublisher.java:283)
	at com.linecorp.armeria.internal.common.stream.SurroundingPublisher$SurroundingSubscriber.publishDownstream(SurroundingPublisher.java:367)
	at com.linecorp.armeria.internal.common.stream.SurroundingPublisher$SurroundingSubscriber.onNext(SurroundingPublisher.java:384)
	at com.linecorp.armeria.internal.common.SplitHttpMessageSubscriber.onNext0(SplitHttpMessageSubscriber.java:239)
	at com.linecorp.armeria.internal.common.SplitHttpMessageSubscriber.lambda$onNext$2(SplitHttpMessageSubscriber.java:228)
	at com.linecorp.armeria.common.DefaultContextAwareRunnable.run(DefaultContextAwareRunnable.java:45)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:166)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
	at --- Async.Stack.Trace --- (captured by IntelliJ IDEA debugger)
	at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java)
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:836)
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute0(SingleThreadEventExecutor.java:827)
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:817)
	at com.linecorp.armeria.common.AbstractContextAwareExecutor.execute(AbstractContextAwareExecutor.java:81)
	at com.linecorp.armeria.internal.common.SplitHttpMessageSubscriber.onNext(SplitHttpMessageSubscriber.java:228)
	at com.linecorp.armeria.internal.common.DefaultSplitHttpResponse$SplitHttpResponseBodySubscriber.onNext(DefaultSplitHttpResponse.java:98)
	at com.linecorp.armeria.internal.common.DefaultSplitHttpResponse$SplitHttpResponseBodySubscriber.onNext(DefaultSplitHttpResponse.java:70)
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriberWithElements(DefaultStreamMessage.java:420)
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriber0(DefaultStreamMessage.java:398)
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.doRequest(DefaultStreamMessage.java:285)
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.request(DefaultStreamMessage.java:267)
	at com.linecorp.armeria.common.stream.CancellableStreamMessage$SubscriptionImpl.request(CancellableStreamMessage.java:215)
	at com.linecorp.armeria.internal.common.SplitHttpMessageSubscriber.request0(SplitHttpMessageSubscriber.java:194)
	at com.linecorp.armeria.internal.common.SplitHttpMessageSubscriber.request(SplitHttpMessageSubscriber.java:183)
	at com.linecorp.armeria.internal.common.stream.SurroundingPublisher$SurroundingSubscriber.requestUpstream(SurroundingPublisher.java:339)
	at com.linecorp.armeria.internal.common.stream.SurroundingPublisher$SurroundingSubscriber.publish(SurroundingPublisher.java:283)
	at com.linecorp.armeria.internal.common.stream.SurroundingPublisher$SurroundingSubscriber.publishDownstream(SurroundingPublisher.java:367)
	at com.linecorp.armeria.internal.common.stream.SurroundingPublisher$SurroundingSubscriber.onNext(SurroundingPublisher.java:384)
	at com.linecorp.armeria.internal.common.SplitHttpMessageSubscriber.onNext0(SplitHttpMessageSubscriber.java:239)
	at com.linecorp.armeria.internal.common.SplitHttpMessageSubscriber.lambda$onNext$2(SplitHttpMessageSubscriber.java:228)
	at com.linecorp.armeria.common.DefaultContextAwareRunnable.run(DefaultContextAwareRunnable.java:45)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:166)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
	at --- Async.Stack.Trace --- (captured by IntelliJ IDEA debugger)
	at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java)
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:836)
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute0(SingleThreadEventExecutor.java:827)
	at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:817)
	at com.linecorp.armeria.common.AbstractContextAwareExecutor.execute(AbstractContextAwareExecutor.java:81)
	at com.linecorp.armeria.internal.common.SplitHttpMessageSubscriber.onNext(SplitHttpMessageSubscriber.java:228)
	at com.linecorp.armeria.internal.common.DefaultSplitHttpResponse$SplitHttpResponseBodySubscriber.onNext(DefaultSplitHttpResponse.java:98)
	at com.linecorp.armeria.internal.common.DefaultSplitHttpResponse$SplitHttpResponseBodySubscriber.onNext(DefaultSplitHttpResponse.java:70)
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriberWithElements(DefaultStreamMessage.java:420)
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriber0(DefaultStreamMessage.java:398)
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriber(DefaultStreamMessage.java:344)
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.addObjectOrEvent(DefaultStreamMessage.java:325)
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.addObject(DefaultStreamMessage.java:252)
	at com.linecorp.armeria.common.stream.AbstractStreamWriter.tryWrite(AbstractStreamWriter.java:64)
	at com.linecorp.armeria.common.stream.DefaultStreamMessage.tryWrite(DefaultStreamMessage.java:70)
	at com.linecorp.armeria.internal.client.DecodedHttpResponse.tryWrite(DecodedHttpResponse.java:54)
	at com.linecorp.armeria.client.HttpResponseWrapper.tryWriteData(HttpResponseWrapper.java:176)
	at com.linecorp.armeria.client.Http2ResponseDecoder.onDataRead(Http2ResponseDecoder.java:278)
	at io.netty.handler.codec.http2.Http2FrameListenerDecorator.onDataRead(Http2FrameListenerDecorator.java:34)
	at io.netty.handler.codec.http2.Http2EmptyDataFrameListener.onDataRead(Http2EmptyDataFrameListener.java:49)
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:320)
	at io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onDataRead(Http2InboundFrameLogger.java:46)
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:409)
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:244)
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:164)
	at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:39)
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:186)
	at io.netty.handler.codec.http2.DecoratingHttp2ConnectionDecoder.decodeFrame(DecoratingHttp2ConnectionDecoder.java:61)
	at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:391)
	at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:451)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1515)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1378)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1427)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.flush.FlushConsolidationHandler.channelRead(FlushConsolidationHandler.java:152)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:501)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:399)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:840)

I revised changelog for this version but found no exact problem.

The implementation that works with previous versions is as follows:

/**
	 * Method streams contents of the single file identified by its unique UUID to the client.
	 */
	@Override
	public void fetchFile(GrpcFetchFileRequest request, StreamObserver<GrpcFetchFileResponse> responseObserver) {
		executeWithClientContext(
			() -> {
				final UUID fileId = toUuid(request.getFileId());
				final Optional<FileForFetch> fileToFetch = this.management.getFileToFetch(fileId);
				if (fileToFetch.isEmpty()) {
					sendErrorToClient(new FileForFetchNotFoundException(fileId), responseObserver);
				} else {
					try (
						final InputStream inputStream = this.management.fetchFile(
							fileId
						)
					) {
						//noinspection CheckForOutOfMemoryOnLargeArrayAllocation
						byte[] buffer = new byte[65_536];
						int bytesRead;
						while ((bytesRead = inputStream.read(buffer)) != -1) {
							GrpcFetchFileResponse response = GrpcFetchFileResponse.newBuilder()
								.setFileContents(ByteString.copyFrom(buffer, 0, bytesRead))
								.setTotalSizeInBytes(fileToFetch.get().totalSizeInBytes())
								.build();
							responseObserver.onNext(response);
						}
						responseObserver.onCompleted();
					} catch (IOException e) {
						throw new UnexpectedIOException(
							"Failed to fetch the designated file: " + e.getMessage(),
							"Failed to fetch the designated file.",
							e
						);
					}
				}
			},
			this.evita.getRequestExecutor(),
			responseObserver,
			this.context
		);
	}

The proto declaration looks like this:

message GrpcFetchFileRequest {
  GrpcUuid fileId = 1;
}

message GrpcFetchFileResponse {
  bytes fileContents = 1;
  int64 totalSizeInBytes = 2;
}
  
rpc FetchFile(GrpcFetchFileRequest) returns (stream GrpcFetchFileResponse);

If you want to see the problem in action (in actual test), here are the steps to reproduce:

1. git clone https://github.com/FgForrest/evitaDB.git
2. git pull armeria-missing-grpc-status-code
3. execute test io.evitadb.driver.EvitaClientReadWriteTest#shouldBackupAndRestoreCatalogViaDownloadingAndUploadingFileContents

If you revert version of Armeria in pom.xml back to 1.32.3, the test correctly passes.

[jrhee17]

Thanks for the detailed analysis. The bug will likely be fixed with the following PR: #6213.

Let me add this to the milestone so that we can track if the issue indeed was fixed in the next release.

[novoj]

Thank you for response, if you'd like to verify the fix, I'd love to do it - it's easy for me.

[ikhoon]

Resolved by #6213

[novoj]

I confirm, that the problem no longer occurs with Armeria 1.33.0. Thanks 😍