[Bug] skywakling hanged up request under multi-concurrent requests on spring cloud gateway

[chengfq5]

Search before asking

• I had searched in the issues and found no similar issues.

Apache SkyWalking Component

Java Agent (apache/skywalking-java)

What happened

skywakling hanged up request under multi-concurrent requests on spring cloud gateway

What you expected to happen

scg handed up request, and can recur stably when assign parameter of "-c " greatter than 1

ab -c 4 -n 400 -T 'application/json;charset=UTF-8' 'http://127.0.0.1:30000/api/hello'
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
apr_pollset_poll: The timeout specified has expired (70007)
Total of 398 requests completed

spring-cloud-gateway throwed excepiton :

2022-11-25 17:10:42.491 ERROR 44595 --- [ctor-http-nio-6] reactor.core.publisher.Operators : Operator called default onErrorDropped

java.lang.NullPointerException: null
at org.apache.skywalking.apm.plugin.spring.cloud.gateway.v20x.HttpClientRequestInterceptor$2.accept(HttpClientRequestInterceptor.java:98) ~[na:na]
at org.apache.skywalking.apm.plugin.spring.cloud.gateway.v20x.HttpClientRequestInterceptor$2.accept(HttpClientRequestInterceptor.java:94) ~[na:na]
at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:189) ~[reactor-core-3.1.10.RELEASE.jar:3.1.10.RELEASE]
at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:201) [reactor-core-3.1.10.RELEASE.jar:3.1.10.RELEASE]
at reactor.core.publisher.FluxRetryPredicate$RetryPredicateSubscriber.onNext(FluxRetryPredicate.java:81) [reactor-core-3.1.10.RELEASE.jar:3.1.10.RELEASE]
at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:146) [reactor-core-3.1.10.RELEASE.jar:3.1.10.RELEASE]
at reactor.ipc.netty.channel.PooledClientContextHandler.fireContextActive(PooledClientContextHandler.java:87) [reactor-netty-0.7.10.RELEASE.jar:0.7.10.RELEASE]
at reactor.ipc.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:584) [reactor-netty-0.7.10.RELEASE.jar:0.7.10.RELEASE]
at reactor.ipc.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:138) [reactor-netty-0.7.10.RELEASE.jar:0.7.10.RELEASE]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.29.Final.jar:4.1.29.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.29.Final.jar:4.1.29.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413) [netty-codec-4.1.29.Final.jar:4.1.29.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [netty-codec-4.1.29.Final.jar:4.1.29.Final]
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:628) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:563) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:480) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) [netty-common-4.1.29.Final.jar:4.1.29.Final]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_301]

2022-11-25 17:10:42.509 ERROR 44595 --- [ctor-http-nio-6] r.ipc.netty.channel.ContextHandler : Error cannot be forwarded to user-facing Mono

reactor.core.Exceptions$BubblingException: java.lang.NullPointerException
at reactor.core.Exceptions.bubble(Exceptions.java:154) ~[reactor-core-3.1.10.RELEASE.jar:3.1.10.RELEASE]
at reactor.core.publisher.Operators.onErrorDropped(Operators.java:263) ~[reactor-core-3.1.10.RELEASE.jar:3.1.10.RELEASE]
at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:193) ~[reactor-core-3.1.10.RELEASE.jar:3.1.10.RELEASE]
at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:201) ~[reactor-core-3.1.10.RELEASE.jar:3.1.10.RELEASE]
at reactor.core.publisher.FluxRetryPredicate$RetryPredicateSubscriber.onNext(FluxRetryPredicate.java:81) ~[reactor-core-3.1.10.RELEASE.jar:3.1.10.RELEASE]
at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:146) ~[reactor-core-3.1.10.RELEASE.jar:3.1.10.RELEASE]
at reactor.ipc.netty.channel.PooledClientContextHandler.fireContextActive(PooledClientContextHandler.java:87) ~[reactor-netty-0.7.10.RELEASE.jar:0.7.10.RELEASE]
at reactor.ipc.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:584) ~[reactor-netty-0.7.10.RELEASE.jar:0.7.10.RELEASE]
at reactor.ipc.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:138) ~[reactor-netty-0.7.10.RELEASE.jar:0.7.10.RELEASE]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.29.Final.jar:4.1.29.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.29.Final.jar:4.1.29.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413) [netty-codec-4.1.29.Final.jar:4.1.29.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [netty-codec-4.1.29.Final.jar:4.1.29.Final]
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:628) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:563) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:480) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) [netty-common-4.1.29.Final.jar:4.1.29.Final]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_301]
Caused by: java.lang.NullPointerException: null
at org.apache.skywalking.apm.plugin.spring.cloud.gateway.v20x.HttpClientRequestInterceptor$2.accept(HttpClientRequestInterceptor.java:98) ~[na:na]
at org.apache.skywalking.apm.plugin.spring.cloud.gateway.v20x.HttpClientRequestInterceptor$2.accept(HttpClientRequestInterceptor.java:94) ~[na:na]
at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:189) ~[reactor-core-3.1.10.RELEASE.jar:3.1.10.RELEASE]
... 29 common frames omitted

How to reproduce

spring-cloud-gateway: 2.0.2.RELEASE
skywalking-version: apache-skywalking-java-agent-8.13.0

test scg demo: https://github.com/chengfq5/scg_demo.git
jvm options: -javaagent:/XXX/skywalking-agent/skywalking-agent.jar

ab -c 4 -n 400 -T 'application/json;charset=UTF-8' 'http://127.0.0.1:30000/api/hello'
/hello is a any api in another servcie need to mock by yourself

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[wu-sheng]

Various plugins are contributed by hundreds of contributors, including this one.

I moved this to the discussion. If anyone wants to do further research, they could pick this up. SkyWalking community can't afford to debug plugins one by one.

I could only provide Gateway 2.0 test case here, https://github.com/apache/skywalking-java/tree/main/test/plugin/scenarios/gateway-2.0.x-scenario. This is how the plugin is tested.