Grpc server thread pool is full, rejecting the task

[Yelijah]

skywalking oap'version is 9.1.1 and the storage backend is elasticsearch.

my skywalking oap can't accept any data sometimes, the logs show like

2022-12-06 17:21:35,881 org.apache.skywalking.oap.server.library.server.grpc.GRPCServer 115 [grpc-default-worker-ELG-7-13] WARN  [] - Grpc server thread pool is full, rejecting the task
2022-12-06 17:21:36,000 org.apache.skywalking.oap.server.library.server.grpc.GRPCServer 115 [grpc-default-worker-ELG-7-13] WARN  [] - Grpc server thread pool is full, rejecting the task
2022-12-06 17:21:36,019 org.apache.skywalking.oap.server.library.server.grpc.GRPCServer 115 [grpc-default-worker-ELG-7-13] WARN  [] - Grpc server thread pool is full, rejecting the task

All grpcServerPool threads are waiting for BlockingQueue, like this:

"grpcServerPool-1-thread-118" #375 prio=5 os_prio=0 tid=0x0000563e09cde800 nid=0x198 waiting on condition [0x00007f589faa3000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x0000000080b89098> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:353)
	at org.apache.skywalking.oap.server.library.datacarrier.buffer.ArrayBlockingQueueBuffer.save(ArrayBlockingQueueBuffer.java:45)
	at org.apache.skywalking.oap.server.library.datacarrier.buffer.Channels.save(Channels.java:58)
	at org.apache.skywalking.oap.server.library.datacarrier.DataCarrier.produce(DataCarrier.java:86)
	at org.apache.skywalking.oap.server.core.analysis.worker.MetricsAggregateWorker.in(MetricsAggregateWorker.java:86)
	at org.apache.skywalking.oap.server.core.analysis.worker.MetricsStreamProcessor.in(MetricsStreamProcessor.java:104)
	at org.apache.skywalking.oap.server.core.analysis.manual.relation.service.ServiceCallRelationDispatcher.clientSide(ServiceCallRelationDispatcher.java:56)
	at org.apache.skywalking.oap.server.core.analysis.manual.relation.service.ServiceCallRelationDispatcher.dispatch(ServiceCallRelationDispatcher.java:34)
	at org.apache.skywalking.oap.server.core.analysis.manual.relation.service.ServiceCallRelationDispatcher.dispatch(ServiceCallRelationDispatcher.java:25)
	at org.apache.skywalking.oap.server.core.analysis.DispatcherManager.forward(DispatcherManager.java:61)
	at org.apache.skywalking.oap.server.core.source.SourceReceiverImpl.receive(SourceReceiverImpl.java:36)
	at org.apache.skywalking.oap.server.analyzer.provider.trace.parser.listener.RPCAnalysisListener.lambda$build$2(RPCAnalysisListener.java:312)
	at org.apache.skywalking.oap.server.analyzer.provider.trace.parser.listener.RPCAnalysisListener$$Lambda$1274/191062474.accept(Unknown Source)
	at java.util.ArrayList.forEach(ArrayList.java:1257)
	at org.apache.skywalking.oap.server.analyzer.provider.trace.parser.listener.RPCAnalysisListener.build(RPCAnalysisListener.java:310)
	at org.apache.skywalking.oap.server.analyzer.provider.trace.parser.TraceAnalyzer$$Lambda$1272/713676582.accept(Unknown Source)
	at java.util.ArrayList.forEach(ArrayList.java:1257)
	at org.apache.skywalking.oap.server.analyzer.provider.trace.parser.TraceAnalyzer.notifyListenerToBuild(TraceAnalyzer.java:75)
	at org.apache.skywalking.oap.server.analyzer.provider.trace.parser.TraceAnalyzer.doAnalysis(TraceAnalyzer.java:71)
	at org.apache.skywalking.oap.server.analyzer.provider.trace.parser.SegmentParserServiceImpl.send(SegmentParserServiceImpl.java:40)
	at org.apache.skywalking.oap.server.receiver.trace.provider.handler.v8.grpc.TraceSegmentReportServiceHandler$1.onNext(TraceSegmentReportServiceHandler.java:72)
	at org.apache.skywalking.oap.server.receiver.trace.provider.handler.v8.grpc.TraceSegmentReportServiceHandler$1.onNext(TraceSegmentReportServiceHandler.java:63)
	at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:262)
	at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:318)
	at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:301)
	at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:834)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

But most consume threads are sleeping, like

"DataCarrier.METRICS_L2_AGGREGATION.BulkConsumePool.7.Thread" #34 daemon prio=5 os_prio=0 tid=0x0000563e10b5f800 nid=0x45 sleeping[0x00007f58bd7f7000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.apache.skywalking.oap.server.library.datacarrier.consumer.MultipleChannelsConsumer.run(MultipleChannelsConsumer.java:57)

"DataCarrier.METRICS_L2_AGGREGATION.BulkConsumePool.6.Thread" #33 daemon prio=5 os_prio=0 tid=0x0000563e10b5e000 nid=0x44 sleeping[0x00007f58bd8f8000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.apache.skywalking.oap.server.library.datacarrier.consumer.MultipleChannelsConsumer.run(MultipleChannelsConsumer.java:57)

"DataCarrier.METRICS_L2_AGGREGATION.BulkConsumePool.5.Thread" #32 daemon prio=5 os_prio=0 tid=0x0000563e10b5c000 nid=0x43 sleeping[0x00007f58bd9f9000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.apache.skywalking.oap.server.library.datacarrier.consumer.MultipleChannelsConsumer.run(MultipleChannelsConsumer.java:57)

"DataCarrier.METRICS_L2_AGGREGATION.BulkConsumePool.4.Thread" #31 daemon prio=5 os_prio=0 tid=0x0000563e0f6fa800 nid=0x42 sleeping[0x00007f58bdafa000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.apache.skywalking.oap.server.library.datacarrier.consumer.MultipleChannelsConsumer.run(MultipleChannelsConsumer.java:57)

I don't understand why this problem happened and how to fix it.

jstack.txt