enhance: add user/segment level task concurrency limit when using user-task-polling

[chyezh]

issue: #46586
pr: #46587

• limit the task concurrency by user.
• limit the segment sq concurrency when cpu is high.
• add more metric to observe the behaviour.

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: chyezh
To complete the pull request process, please assign zhengbuqian after the PR has been reviewed.
You can assign the PR to them by writing /assign @zhengbuqian in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sre-ci-robot]

[ci-v2-notice]
Notice: New ci-v2 system is enabled for this PR.

To rerun ci-v2 checks, comment with:

• /ci-rerun-code-check // for ci-v2/code-check
• /ci-rerun-build // for ci-v2/build
• /ci-rerun-ut-integration // for ci-v2/ut-integration, will rerun ci-v2/build
• /ci-rerun-ut-go // for ci-v2/ut-go, will rerun ci-v2/build
• /ci-rerun-ut-cpp // for ci-v2/ut-cpp
• /ci-rerun-ut // for all ci-v2/ut-integration, ci-v2/ut-go, ci-v2/ut-cpp, will rerun ci-v2/build
• /ci-rerun-e2e-arm // for ci-v2/e2e-arm
• /ci-rerun-e2e-default // for ci-v2/e2e-default

If you have any questions or requests, please contact @zhikunyao.

[sre-ci-robot]

[INFO] PR Label Summary by Default
[FAILED] PR #46587 not merged

• Title: enhance: add user/segment level task concurrency limit when using user-task-polling
• Status: OPEN
• Link: enhance: add user/segment level task concurrency limit when using user-task-polling #46587

Use /refresh-label to update related check and label manually

[codecov]

Codecov Report

❌ Patch coverage is 82.79570% with 48 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.55%. Comparing base (67e4a21) to head (ab0dffa).
⚠️ Report is 26 commits behind head on 2.6.

Files with missing lines	Patch %	Lines
...querynodev2/segments/limiter/segment_sq_limiter.go	75.00%	20 Missing and 7 partials ⚠️
...l/searchutil/scheduler/user_task_polling_policy.go	74.57%	9 Missing and 6 partials ⚠️
internal/util/searchutil/scheduler/queues.go	50.00%	2 Missing and 1 partial ⚠️
pkg/util/merr/utils.go	0.00%	2 Missing ⚠️
internal/util/searchutil/scheduler/tasks.go	85.71%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##              2.6   #46729      +/-   ##
==========================================
+ Coverage   77.00%   78.55%   +1.54%     
==========================================
  Files        1380     1903     +523     
  Lines      216330   298359   +82029     
==========================================
+ Hits       166577   234364   +67787     
- Misses      42298    56526   +14228     
- Partials     7455     7469      +14     

Components	Coverage Δ
Client	78.17% <ø> (ø)
Core	82.72% <ø> (∅)
Go	77.79% <82.79%> (-0.03%)	⬇️

Files with missing lines	Coverage Δ
internal/querynodev2/segments/search.go	57.98% <100.00%> (+0.50%)	⬆️
internal/querynodev2/segments/segment_do.go	50.00% <100.00%> (+13.41%)	⬆️
pkg/metrics/hardware_metrics.go	100.00% <100.00%> (ø)
pkg/metrics/metrics.go	100.00% <100.00%> (ø)
pkg/metrics/querynode_metrics.go	100.00% <100.00%> (ø)
pkg/util/hardware/listener.go	94.33% <100.00%> (+1.31%)	⬆️
pkg/util/paramtable/component_param.go	97.66% <100.00%> (+0.02%)	⬆️
internal/util/searchutil/scheduler/tasks.go	66.66% <85.71%> (+8.77%)	⬆️
pkg/util/merr/utils.go	79.86% <0.00%> (-0.19%)	⬇️
internal/util/searchutil/scheduler/queues.go	95.65% <50.00%> (-2.11%)	⬇️
... and 2 more

... and 545 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.