[Bug] In Dubbo 3.3.0 & 3.2.16, when using AsyncContext for asynchronous service processing, during a loop of 10,000 iterations, a java.util.concurrent.RejectedExecutionException: null is occasionally thrown.

[judong]

Pre-check

• I am sure that all the content I provide is in English.

Search before asking

• I had searched in the issues and found no similar issues.

Apache Dubbo Component

Java SDK (apache/dubbo)

Dubbo Version

This issue occurs in both Dubbo 3.3.0 and 3.2.16.
Java version: 17.0.12
Operating System: macOS 14.6.1 (23G93)

Steps to reproduce this issue

When using AsyncContext for asynchronous service processing, a java.util.concurrent.RejectedExecutionException: null is occasionally thrown during a loop of 10,000 iterations.

However, the same code works successfully without errors in Dubbo 3.2.15.

2024-09-14T14:58:17.101+04:00  WARN 39176 --- [lientWorker-2-1] r.p.t.t.TripleHttp2ClientResponseHandler :  [DUBBO] Meet Exception on ClientResponseHandler, status code is: INTERNAL, dubbo version: 3.3.0, current host: 192.168.200.145, error code: 4-10. This may be caused by , go to https://dubbo.apache.org/faq/4/10 to find instructions. 

java.util.concurrent.RejectedExecutionException: null
	at org.apache.dubbo.common.threadpool.ThreadlessExecutor.execute(ThreadlessExecutor.java:101) ~[dubbo-3.3.0.jar:3.3.0]
	at org.apache.dubbo.common.threadpool.serial.SerializingExecutor.schedule(SerializingExecutor.java:73) ~[dubbo-3.3.0.jar:3.3.0]
	at org.apache.dubbo.common.threadpool.serial.SerializingExecutor.execute(SerializingExecutor.java:65) ~[dubbo-3.3.0.jar:3.3.0]
	at org.apache.dubbo.rpc.protocol.tri.stream.AbstractTripleClientStream$ClientTransportListener.onClose(AbstractTripleClientStream.java:459) ~[dubbo-3.3.0.jar:3.3.0]
	at org.apache.dubbo.rpc.protocol.tri.transport.TripleHttp2ClientResponseHandler.channelInactive(TripleHttp2ClientResponseHandler.java:83) ~[dubbo-3.3.0.jar:3.3.0]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303) ~[netty-transport-4.1.112.Final.jar:4.1.112.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) ~[netty-transport-4.1.112.Final.jar:4.1.112.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274) ~[netty-transport-4.1.112.Final.jar:4.1.112.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1402) ~[netty-transport-4.1.112.Final.jar:4.1.112.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:301) ~[netty-transport-4.1.112.Final.jar:4.1.112.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) ~[netty-transport-4.1.112.Final.jar:4.1.112.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:900) ~[netty-transport-4.1.112.Final.jar:4.1.112.Final]
	at io.netty.handler.codec.http2.AbstractHttp2StreamChannel$Http2ChannelUnsafe$2.run(AbstractHttp2StreamChannel.java:797) ~[netty-codec-http2-4.1.112.Final.jar:4.1.112.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.runTask$$$capture(AbstractEventExecutor.java:173) ~[netty-common-4.1.112.Final.jar:4.1.112.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java) ~[netty-common-4.1.112.Final.jar:4.1.112.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:166) ~[netty-common-4.1.112.Final.jar:4.1.112.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java) ~[netty-common-4.1.112.Final.jar:4.1.112.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) ~[netty-common-4.1.112.Final.jar:4.1.112.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566) ~[netty-transport-4.1.112.Final.jar:4.1.112.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994) ~[netty-common-4.1.112.Final.jar:4.1.112.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.112.Final.jar:4.1.112.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.112.Final.jar:4.1.112.Final]
	at java.base/java.lang.Thread.run(Thread.java:840) ~[na:na]

What you expected to happen

3.3.0 version and 3.2.16 versin dubbo, not this error

Anything else

No response

Are you willing to submit a pull request to fix on your own?

• Yes I am willing to submit a pull request on my own!

Code of Conduct

• I agree to follow this project's Code of Conduct

[AlbumenJ]

You may exceed the concurrency limit in provider side

[judong]

You may exceed the concurrency limit in provider side

but... 3.2.15 not this problem

[AlbumenJ]

Can you please provide a demo?

[the3rd]

I got the same error and roll back to 3.2.15.

[AllKai]

I got the same error and version is 3.3.1

[AllKai]

java.util.concurrent.RejectedExecutionException: null
	at org.apache.dubbo.common.threadpool.ThreadlessExecutor.execute(ThreadlessExecutor.java:101) ~[dubbo-3.3.1.jar!/:3.3.1]
	at org.apache.dubbo.common.threadpool.serial.SerializingExecutor.schedule(SerializingExecutor.java:73) ~[dubbo-3.3.1.jar!/:3.3.1]
	at org.apache.dubbo.common.threadpool.serial.SerializingExecutor.execute(SerializingExecutor.java:65) ~[dubbo-3.3.1.jar!/:3.3.1]
	at org.apache.dubbo.rpc.protocol.tri.stream.AbstractTripleClientStream$ClientTransportListener.onClose(AbstractTripleClientStream.java:459) ~[dubbo-3.3.1.jar!/:3.3.1]
	at org.apache.dubbo.rpc.protocol.tri.transport.TripleHttp2ClientResponseHandler.channelInactive(TripleHttp2ClientResponseHandler.java:83) ~[dubbo-3.3.1.jar!/:3.3.1]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303) ~[netty-transport-4.1.112.Final.jar!/:4.1.112.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) ~[netty-transport-4.1.112.Final.jar!/:4.1.112.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274) ~[netty-transport-4.1.112.Final.jar!/:4.1.112.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1402) ~[netty-transport-4.1.112.Final.jar!/:4.1.112.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:301) ~[netty-transport-4.1.112.Final.jar!/:4.1.112.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) ~[netty-transport-4.1.112.Final.jar!/:4.1.112.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:900) ~[netty-transport-4.1.112.Final.jar!/:4.1.112.Final]
	at io.netty.handler.codec.http2.AbstractHttp2StreamChannel$Http2ChannelUnsafe$2.run(AbstractHttp2StreamChannel.java:797) ~[netty-codec-http2-4.1.112.Final.jar!/:4.1.112.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.112.Final.jar!/:4.1.112.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.112.Final.jar!/:4.1.112.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) ~[netty-common-4.1.112.Final.jar!/:4.1.112.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566) ~[netty-transport-4.1.112.Final.jar!/:4.1.112.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994) ~[netty-common-4.1.112.Final.jar!/:4.1.112.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.112.Final.jar!/:4.1.112.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.112.Final.jar!/:4.1.112.Final]
	at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]

[AllKai]

java.util.concurrent.RejectedExecutionException: null
	at org.apache.dubbo.common.threadpool.ThreadlessExecutor.execute(ThreadlessExecutor.java:101) ~[dubbo-3.3.1.jar!/:3.3.1]
	at org.apache.dubbo.common.threadpool.serial.SerializingExecutor.schedule(SerializingExecutor.java:73) ~[dubbo-3.3.1.jar!/:3.3.1]
	at org.apache.dubbo.common.threadpool.serial.SerializingExecutor.execute(SerializingExecutor.java:65) ~[dubbo-3.3.1.jar!/:3.3.1]
	at org.apache.dubbo.rpc.protocol.tri.stream.AbstractTripleClientStream$ClientTransportListener.onClose(AbstractTripleClientStream.java:459) ~[dubbo-3.3.1.jar!/:3.3.1]
	at org.apache.dubbo.rpc.protocol.tri.transport.TripleHttp2ClientResponseHandler.channelInactive(TripleHttp2ClientResponseHandler.java:83) ~[dubbo-3.3.1.jar!/:3.3.1]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303) ~[netty-transport-4.1.112.Final.jar!/:4.1.112.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) ~[netty-transport-4.1.112.Final.jar!/:4.1.112.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274) ~[netty-transport-4.1.112.Final.jar!/:4.1.112.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1402) ~[netty-transport-4.1.112.Final.jar!/:4.1.112.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:301) ~[netty-transport-4.1.112.Final.jar!/:4.1.112.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281) ~[netty-transport-4.1.112.Final.jar!/:4.1.112.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:900) ~[netty-transport-4.1.112.Final.jar!/:4.1.112.Final]
	at io.netty.handler.codec.http2.AbstractHttp2StreamChannel$Http2ChannelUnsafe$2.run(AbstractHttp2StreamChannel.java:797) ~[netty-codec-http2-4.1.112.Final.jar!/:4.1.112.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.112.Final.jar!/:4.1.112.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.112.Final.jar!/:4.1.112.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) ~[netty-common-4.1.112.Final.jar!/:4.1.112.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566) ~[netty-transport-4.1.112.Final.jar!/:4.1.112.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994) ~[netty-common-4.1.112.Final.jar!/:4.1.112.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.112.Final.jar!/:4.1.112.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.112.Final.jar!/:4.1.112.Final]
	at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]

[AlbumenJ]

Client is already timeout and throw reject execution exception

[AllKai]

Client is already timeout and throw reject execution exception

So, how can I solve this?

[AlbumenJ]

Check if your load is really high or change the timeout

[the3rd]

Check if your load is really high or change the timeout

3.2.15 is OK. Why?