[Enhancement]: [streaming] Standalone cpu usage is high during concurrent inserts

[ThreadDao]

Is there an existing issue for this?

• I have searched the existing issues

What would you like to be added?

server

• image: chyezh-enhance_make_recovery_components_full_debug-5ed1fe0-20250508
• standalone: 2c8g
• mq: woodpecker with local
• config

    woodpecker:
      logstore:
        fragmentManager:
          maxInterval: 200 
      storage:
        type: local

test

1. create a collection with 2 fields: 1 int64 pk + 1 768im float vector
2. insert data with batch 100 concurrent 30

results

• cpu usage
  metrics link
  

Why is this needed?

No response

Anything else?

No response

[ThreadDao]

from @chyezh Sorting does not have high CPU overhead. However, pq generates binlog with zstd enabled, which has high CPU overhead. FLUSH+SORT is used to regenerate the log. The write amplification of binlog storage is quite serious. For standalone, there should be a lot of room for optimization.

[xiaofan-luan]

guess there is nothing we can do?

CPU is simply consumed while write.
StorageV2 might be better but so far this might not be critical.

Did we compare 2.5 and latest master?

[chyezh]

guess there is nothing we can do?

CPU is simply consumed while write. StorageV2 might be better but so far this might not be critical.

Did we compare 2.5 and latest master?

@xiaofan-luan
We can use storagev2 to verify it.
In these case, the pqarrow apply memory operation.
zstd is also use much cpu.
Milvus's stats+flush generate some overhead, especially in standalone mode.
I think it's a good topic to optimize even if we use storagev2.

[yanliang567]

/assign @liliu-z

[xiaofan-luan]

@ThreadDao
can we compare storage v1 and v2 in this case see if there are some difference?

[chyezh]

related pr: #42630

[liliu-z]

/assign @ThreadDao