I often encounter this error in my logs: ClosedStreamException: received a RST_stREAM frame: CANCEL

[liuhao710118]

skywalking v9.7
elasticsearch：v8.14

I want to know what caused this error. I can't find any relevant information online, so I would like to inquire here

Here are the key parts of my log

2024-08-29 16:30:19,455 - com.linecorp.armeria.server.logging.LoggingService - 95 [armeria-eventloop-epoll-5-2] WARN  [] - [sreqId=b841785b, chanId=82ef2569, raddr=172.16.251.172:60706, laddr=172.16.251.172:12800][h2c://localhost.localdomain/internal/l7check#HEAD] Request: {startTime=2024-08-29T08:29:28.800Z(1724920168800483), length=0B, duration=50579ms(50579404976ns), scheme=none+h2c, name=HEAD, headers=[EOS, :method=HEAD, :authority=172.16.251.172:12800, :scheme=http, :path=/internal/l7check, if-none-match="healthy", prefer=wait=60, user-agent=armeria/1.25.0]}
2024-08-29 16:30:19,455 - com.linecorp.armeria.server.logging.LoggingService - 156 [armeria-eventloop-epoll-5-2] WARN  [] - [sreqId=b841785b, chanId=82ef2569, raddr=172.16.251.172:60706, laddr=172.16.251.172:12800][h2c://localhost.localdomain/internal/l7check#HEAD] Response: {startTime=2024-08-29T08:29:28.800Z(1724920168800742), length=0B, duration=50579ms(50579672744ns), totalDuration=50579ms(50579933069ns), cause=com.linecorp.armeria.common.stream.ClosedStreamException: received a RST_STREAM frame: CANCEL, headers=[:status=0]}
com.linecorp.armeria.common.stream.ClosedStreamException: received a RST_STREAM frame: CANCEL
	at com.linecorp.armeria.server.Http2RequestDecoder.onRstStreamRead(Http2RequestDecoder.java:396) [armeria-1.25.0.jar:?]
	at io.netty.handler.codec.http2.Http2FrameListenerDecorator.onRstStreamRead(Http2FrameListenerDecorator.java:59) [netty-codec-http2-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.handler.codec.http2.Http2MaxRstFrameListener.onRstStreamRead(Http2MaxRstFrameListener.java:56) [netty-codec-http2-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.handler.codec.http2.Http2FrameListenerDecorator.onRstStreamRead(Http2FrameListenerDecorator.java:59) [netty-codec-http2-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onRstStreamRead(DefaultHttp2ConnectionDecoder.java:442) [netty-codec-http2-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onRstStreamRead(Http2InboundFrameLogger.java:80) [netty-codec-http2-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readRstStreamFrame(DefaultHttp2FrameReader.java:509) [netty-codec-http2-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:259) [netty-codec-http2-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:159) [netty-codec-http2-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41) [netty-codec-http2-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:173) [netty-codec-http2-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.handler.codec.http2.DecoratingHttp2ConnectionDecoder.decodeFrame(DecoratingHttp2ConnectionDecoder.java:63) [netty-codec-http2-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.handler.codec.http2.DecoratingHttp2ConnectionDecoder.decodeFrame(DecoratingHttp2ConnectionDecoder.java:63) [netty-codec-http2-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:393) [netty-codec-http2-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453) [netty-codec-http2-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) [netty-codec-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) [netty-codec-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) [netty-codec-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.handler.flush.FlushConsolidationHandler.channelRead(FlushConsolidationHandler.java:152) [netty-handler-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800) [netty-transport-classes-epoll-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:509) [netty-transport-classes-epoll-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407) [netty-transport-classes-epoll-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.100.Final.jar:4.1.100.Final]
	at java.lang.Thread.run(Thread.java:829) [?:?]

[wu-sheng]

This usually means connection disconnected from the client side somehow.

[wu-sheng]

I can't see much different. Maybe your database(I don't know which you chose) is slow or something.

[xbchen711]

I found that when I query indicator data for 7 days or more, this exception occurs, but when I query data for 3 days or less, this exception does not occur

When I query the indicator data interface "/graphql" for 7 days or more, the 503 service is unavailable

I meet the same error, resulting graphql 503 response, any ideas to resolve this problem?

[liuhao710118]

I haven't found a way to deal with this problem.

[ahseezre]

Did you finally solve this problem?

[liuhao710118]

The reason for this error is that the size of the Elasticsearch cluster is too small. You can try increasing the running memory of the Elasticsearch node and adding more cluster nodes. Even so, this problem may still occur. You can download the source code of Skywalking and change the default request duration. The default is 15 seconds. If no result is obtained within 15 seconds, the page side will automatically abort the request. You can increase this parameter

[JackGxd]

i find in application.yml file,when 'cluster' config error while throw this exception.please check SW_CLUSTER and detail