[BUG] Performance Degradation: Pixiu Causes Significant Latency Increase in Dubbo / gRPC / Triple Benchmarks

[Alanxtl]

🚨 Performance Degradation: Pixiu Causes Significant Latency Increase in Dubbo / gRPC / Triple Benchmarks

When benchmarking the gateway performance of dubbo-go-pixiu, I observed significant latency increases across Dubbo, gRPC, and Triple protocols when traffic is routed through Pixiu.

The degradation is especially severe for Triple , where the latency increases by 40–50x ompared to direct invocation.

This appears to indicate a potential bottleneck in Pixiu’s protocol handling, filter chain, connection management, or serialization logic.

Benchmark Results

• dubbo:

dubbo protocol performance test
      Name                 | N   | Min   | Median | Mean  | StdDev | Max   
      =====================================================================
      GetUser [duration]   | 500 | 100µs | 200µs  | 600µs | 2.8ms  | 20.7ms
      ---------------------------------------------------------------------
      GetGender [duration] | 500 | 0s    | 100µs  | 200µs | 100µs  | 600µs 
      ---------------------------------------------------------------------
      GetUser0 [duration]  | 500 | 100µs | 100µs  | 200µs | 0s     | 400µs 
      ---------------------------------------------------------------------
      GetUsers [duration]  | 500 | 100µs | 200µs  | 200µs | 100µs  | 600µs 
      ---------------------------------------------------------------------
      GetUser2 [duration]  | 500 | 0s    | 200µs  | 200µs | 100µs  | 500µs 
      ---------------------------------------------------------------------
      GetErr [duration]    | 500 | 300µs | 500µs  | 500µs | 200µs  | 1.4ms 

 pixiu to dubbo protocol performance test
      Name                 | N   | Min   | Median | Mean  | StdDev | Max  
      ====================================================================
      GetUser [duration]   | 500 | 200µs | 600µs  | 1.8ms | 8.4ms  | 61ms 
      --------------------------------------------------------------------
      GetGender [duration] | 499 | 200µs | 500µs  | 600µs | 300µs  | 4.1ms
      --------------------------------------------------------------------
      GetUser0 [duration]  | 497 | 200µs | 500µs  | 600µs | 300µs  | 2.5ms
      --------------------------------------------------------------------
      GetUsers [duration]  | 495 | 200µs | 600µs  | 700µs | 300µs  | 2.9ms

• grpc:

grpc protocol performance test
      Name                     | N   | Min   | Median | Mean  | StdDev | Max  
      ========================================================================
      GetUser [duration]       | 500 | 100µs | 300µs  | 400µs | 600µs  | 5.2ms
      ------------------------------------------------------------------------
      GetUsers [duration]      | 500 | 100µs | 300µs  | 300µs | 200µs  | 1.5ms
      ------------------------------------------------------------------------
      GetUserByName [duration] | 496 | 100µs | 200µs  | 300µs | 100µs  | 1ms  

pixiu to grpc protocol performance test
      Name                     | N   | Min   | Median | Mean  | StdDev | Max   
      =========================================================================
      GetUser [duration]       | 500 | 600µs | 2ms    | 3.4ms | 4.3ms  | 25.9ms
      -------------------------------------------------------------------------
      GetUsers [duration]      | 500 | 600µs | 1.3ms  | 2.1ms | 3.3ms  | 25.1ms
      -------------------------------------------------------------------------
      GetUserByName [duration] | 500 | 800µs | 1.9ms  | 2.7ms | 3.8ms  | 33.4ms

• triple:

triple protocol performance test
      Name                | N   | Min   | Median | Mean  | StdDev | Max  
      ===================================================================
      SayHello [duration] | 500 | 100µs | 200µs  | 300µs | 300µs  | 2.6ms

pixiu to triple protocol performance test
      Name                | N   | Min    | Median | Mean | StdDev | Max   
      ====================================================================
      SayHello [duration] | 490 | 10.1ms | 12.7ms | 14ms | 4.1ms  | 43.3ms

How to Reproduce

refer to tools/benchmark

you can try

--pprof
--cpu-profile
--mem-profile

检查以下问题：

1. goroutine 阻塞
2. JSON 编码/解码热点
3. 正则匹配/插件 filter 配置开销
4. 连接池复用情况

[Similarityoung]

这个 dubbo 和 triple 都是 v2 的版本了，需要考虑维护提升吗，还是说等 dubbo-go 那弄完直接整最新的泛化调用？

[Alanxtl]

这个 dubbo 和 triple 都是 v2 的版本了，需要考虑维护提升吗，还是说等 dubbo-go 那弄完直接整最新的泛化调用？

主要是从benchmark来看，现在pixiu的性能很差，需要弄清楚是因为什么原因，要不以后弄新的性能也提不上去

[nanjiek]

我现在对于dubbo_suite.test中的pixiu to dubbo protocol performance test采样，排除了JSON的热点问题

[nanjiek]

[nanjiek]

这是 block profile（阻塞分析）的结果

一、核心阻塞节点解读

函数	阻塞时长	占比	含义
runtime.selectgo	31.54s	69.44%	协程在 select 语句中等待通道操作（如接收/发送通道数据）时的阻塞
runtime.chanrecvl	13.67s	30.09%	协程从通道接收数据（<-chan）时，通道无数据可接收导致的阻塞

通用实例池的锁竞争 + 配置过小，导致 futex/selectgo 阻塞；
Dubbo 异步调用的通道接收阻塞，导致 chanrecvl 占比高