enhance: Optimize shard serviceable mechanism #41937
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sre-ci-robot
added
size/XXL
mergify
bot
added
dco-passed
weiliu1031
force-pushed
the
optimize_shard_serviceable
branch
from
e91be31 to
cb3de2d
Compare
|
@weiliu1031 go-sdk check failed, comment |
|
@weiliu1031 E2e jenkins job failed, comment |
weiliu1031
force-pushed
the
optimize_shard_serviceable
branch
from
cb3de2d to
17973bd
Compare
| serviceable := checkDelegatorServiceable(ctx, dh, dmChannel.View) | ||
| // trigger pull next target until shard leader is ready | ||
| if !serviceable { | ||
| dh.lastUpdateTs = 0 |
There was a problem hiding this comment.
lastUpdateTs is not guarded by lock
There was a problem hiding this comment.
each querynode has it's own dist_handler, and dist_handler pull data distribution in serial, the lock is unnecessary
| zap.String("channel", view.Channel), | ||
| zap.Error(err)) | ||
| if status := view.Status; status != nil { | ||
| if !status.GetServiceable() { |
There was a problem hiding this comment.
no need to GetServiceable twice
weiliu1031
force-pushed
the
optimize_shard_serviceable
branch
from
17973bd to
1f7d364
Compare
|
@weiliu1031 E2e jenkins job failed, comment |
- Merge leader view and channel management into ChannelDistManager, allowing a channel to have multiple delegators. - Improve shard leader switching to ensure a single replica only has one shard leader per channel. The shard leader handles all resource loading and query requests. - Refine the serviceable mechanism: after QC completes loading, sync the query view to the delegator. The delegator then determines its serviceable status based on the query view. - When a delegator encounters forwarding query or deletion failures, mark the corresponding segment as offline and transition it to an unserviceable state. Signed-off-by: Wei Liu <[email protected]>
weiliu1031
force-pushed
the
optimize_shard_serviceable
branch
from
1f7d364 to
82eac09
Compare
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: weiliu1031, xiaofan-luan The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
weiliu1031
added a commit
to weiliu1031/milvus
that referenced
this pull request
May 30, 2025
issue: milvus-io#42098 milvus-io#42404 related to: #milvus-io#42009 milvus-io#41937 Implement new method to handle partition removal from next target without directly modifying current target. Changes include: - Add RemovePartitionFromNextTarget method and deprecate RemovePartition - Update target_observer to use new method for ReleasePartition operations - Add unit tests and mock methods for new functionality This ensures that all changes to next target will propagates to delegator's query view. Signed-off-by: Wei Liu <[email protected]>
weiliu1031
added a commit
to weiliu1031/milvus
that referenced
this pull request
May 30, 2025
issue: milvus-io#42098 milvus-io#42404 related to: #milvus-io#42009 milvus-io#41937 Implement new method to handle partition removal from next target without directly modifying current target. Changes include: - Add RemovePartitionFromNextTarget method and deprecate RemovePartition - Update target_observer to use new method for ReleasePartition operations - Add unit tests and mock methods for new functionality This ensures that all changes to next target will propagates to delegator's query view. Signed-off-by: Wei Liu <[email protected]>
weiliu1031
added a commit
to weiliu1031/milvus
that referenced
this pull request
Jun 3, 2025
issue: milvus-io#42098 milvus-io#42404 related to: #milvus-io#42009 milvus-io#41937 Implement new method to handle partition removal from next target without directly modifying current target. Changes include: - Add RemovePartitionFromNextTarget method and deprecate RemovePartition - Update target_observer to use new method for ReleasePartition operations - Add unit tests and mock methods for new functionality This ensures that all changes to next target will propagates to delegator's query view. Signed-off-by: Wei Liu <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
issue: #41690