Milvus Index Check Failure: Unable to Retrieve Index Information for Multiple Segments

[deku0818]

milvusdb/milvus:v2.4.5
standalone

Hello,

I've encountered an issue with Milvus where multiple segments are failing to retrieve index information. The relevant log entries are as follows:

[2024/08/31 14:20:12.979 +00:00] [INFO] [datacoord/services.go:835] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=9f4e22e16a1ec24a73b66d76be673494] [collectionID=451563326804270826] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_1_451563326804270826v0] ["# of unflushed segments"=0] ["# of flushed segments"=4] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0]
[2024/08/31 14:20:12.979 +00:00] [WARN] [checkers/index_checker.go:127] ["failed to get indexInfo for segment"] [collectionID=451563326802326897] [segmentID=451679503151096609] [error="index not found[segmentID=451679503151096609]"]
[2024/08/31 14:20:12.979 +00:00] [WARN] [checkers/index_checker.go:127] ["failed to get indexInfo for segment"] [collectionID=451563326802326897] [segmentID=451679503147719150] [error="index not found[segmentID=451679503147719150]"]
[2024/08/31 14:20:12.980 +00:00] [WARN] [checkers/index_checker.go:127] ["failed to get indexInfo for segment"] [collectionID=451563326802326897] [segmentID=451679503151866997] [error="index not found[segmentID=451679503151866997]"]
[2024/08/31 14:20:12.980 +00:00] [WARN] [checkers/index_checker.go:127] ["failed to get indexInfo for segment"] [collectionID=451563326802326897] [segmentID=451679503151439932] [error="index not found[segmentID=451679503151439932]"]
[2024/08/31 14:20:12.980 +00:00] [WARN] [checkers/index_checker.go:127] ["failed to get indexInfo for segment"] [collectionID=451563326802326897] [segmentID=451679503147663213] [error="index not found[segmentID=451679503147663213]"]
[2024/08/31 14:20:12.981 +00:00] [WARN] [checkers/index_checker.go:127] ["failed to get indexInfo for segment"] [collectionID=451563326802326897] [segmentID=451679503148118356] [error="index not found[segmentID=451679503148118356]"]
[2024/08/31 14:20:12.982 +00:00] [WARN] [checkers/index_checker.go:127] ["failed to get indexInfo for segment"] [collectionID=451563326802326897] [segmentID=451679503154069737] [error="index not found[segmentID=451679503154069737]"]
[2024/08/31 14:20:12.982 +00:00] [WARN] [checkers/index_checker.go:127] ["failed to get indexInfo for segment"] [collectionID=451563326802326897] [segmentID=451679503145459812] [error="index not found[segmentID=451679503145459812]"]
[2024/08/31 14:20:12.982 +00:00] [WARN] [checkers/index_checker.go:127] ["failed to get indexInfo for segment"] [collectionID=451563326802326897] [segmentID=451679503148624285] [error="index not found[segmentID=451679503148624285]"]
[2024/08/31 14:20:12.983 +00:00] [WARN] [checkers/index_checker.go:127] ["failed to get indexInfo for segment"] [collectionID=451563326802326897] [segmentID=451679503149232035] [error="index not found[segmentID=451679503149232035]"]
[2024/08/31 14:20:12.983 +00:00] [WARN] [checkers/index_checker.go:127] ["failed to get indexInfo for segment"] [collectionID=451563326802326897] [segmentID=451679503146794086] [error="index not found[segmentID=451679503146794086]"]
[2024/08/31 14:20:12.983 +00:00] [INFO] [datacoord/index_service.go:924] ["List index success"] [traceID=45bc878de5dfcc5dce4579f85f974f8f] [collectionID=451563326804270826]
[2024/08/31 14:20:12.984 +00:00] [WARN] [checkers/index_checker.go:127] ["failed to get indexInfo for segment"] [collectionID=451563326804270826] [segmentID=451679503154416071] [error="index not found[segmentID=451679503154416071]"]
[2024/08/31 14:20:12.984 +00:00] [WARN] [checkers/index_checker.go:127] ["failed to get indexInfo for segment"] [collectionID=451563326804270826] [segmentID=451679503154879224] [error="index not found[segmentID=451679503154879224]"]
[2024/08/31 14:20:12.984 +00:00] [WARN] [checkers/index_checker.go:127] ["failed to get indexInfo for segment"] [collectionID=451563326804270826] [segmentID=451679503153506922] [error="index not found[segmentID=451679503153506922]"]
[2024/08/31 14:20:12.985 +00:00] [INFO] [datacoord/index_service.go:924] ["List index success"] [traceID=82550ffd190319b10d827679b439d09f] [collectionID=451679503147143198]
[2024/08/31 14:20:12.985 +00:00] [INFO] [datacoord/index_service.go:924] ["List index success"] [traceID=c3915489f9dc741805d45df454ac0902] [collectionID=451679503148256638]
[2024/08/31 14:20:12.985 +00:00] [WARN] [checkers/index_checker.go:127] ["failed to get indexInfo for segment"] [collectionID=451679503148256638] [segmentID=451679503149021726] [error="index not found[segmentID=451679503149021726]"]
[2024/08/31 14:20:12.985 +00:00] [WARN] [checkers/index_checker.go:127] ["failed to get indexInfo for segment"] [collectionID=451679503148256638] [segmentID=451679503149021783] [error="index not found[segmentID=451679503149021783]"]
[2024/08/31 14:20:12.986 +00:00] [INFO] [datacoord/index_servic

It appears that despite the indexes being created for the collections, the system is unable to retrieve index information for several segments. This is indicated by the repeated "index not found" errors.

What I've Checked:
The collections have indexes created, and a check on the collection's index status returns true.
However, the logs still indicate issues with certain segments.
Questions:
What could be causing these specific segments to fail in retrieving index information, despite the collection showing that indexes are present?
What steps can I take to resolve these "index not found" errors and ensure that all segments are correctly indexed and accessible?
Any guidance or suggestions would be greatly appreciated!

Thank you in advance for your help.

[xiaofan-luan]

index build only build all the existing data in the collection.

If you keep inserting data , and don't have enough indexnode in the background, then you can found some of the segment with index.

This won't affect your availbility, but may cause search to be slow.

I would recommend you to add more indexnode, and check the indexnode see if there are some warnning/error logs.

[deku0818]

Thank you!
Do I need to switch from standalone to cluster mode?
If so, how can I deploy a Milvus cluster using Docker? I only found Kubernetes deployment methods in the documentation.

[xiaofan-luan]

1. you can. not deploy cluster version without docker.
2. the easiest way to migrate a cluster is to backup and restore to a new deployment