MilvusClientException: no available shard leaders: channel not available

[Siddeshvr]

Environment
Milvus version: v2.6.12
Deployment mode(standalone or cluster): cluster

When we restart the MilvusDB instance to update any configurations like (CPU, RAM, Woodpecker configs), I have observed that MilvusDB is going down due to crash and below are the logs observed

Client logs :
Caused by: io.milvus.v2.exception.MilvusClientException: no available shard leaders: channel not available[channel=by-dev-rootcoord-dml_1_464388135381283799v7] ErrorCode: SERVER_ERROR, ServerCode: 503

Once again if all the nodes of the MilvusDB are restarted, the error logs stopped and service started working.
Why do we see this issue and during restart why MilvusDB is not able to restart all the nodes properly into the stable state ?
Could you please help me to check this issue

[yhmo]

"no available shard leaders: channel not available" -------------- You might see this error if a querynode(acts as a shard leader) crashes or restarts. Milvus recovers after a query node crashes and cannot work for a few minutes, see the similar issue: #44521. Usually, OOM and defects are the major reasons for query node crashes, "no available shard leaders: channel not available" is a result, not the root cause.

[Siddeshvr]

@yhmo @xiaofan-luan

1. Why is this behavior related to StreamingNode and ETCD ?
2. How will it resolve this issue if I use NVMe disk for ETCD ?
3. Why did Milvus recovered just after I restart all the nodes again ?
4. Do I need to follow any order while deploying Milvus using Helm ?
5. Do I need to perform any post deployment steps ?

Could you please help me to understand this

[yhmo]

In milvus, each collection has a shards_number, if shards_number=2, inserted data is split into two pieces and passed to a message queue(pulsar/kafka/woodpecker). By default, shards_number is 1.
Each shard is handled by a streaming node, the streaming node consumes data from the message queue, and maintains a growing segment in memory for the streaming data. Once the size of a growing segment exceeds a threshold, the data node will persist it into storage as sealed segment. Once a growing segment is converted to a sealed segment, it is evicted from the streaming node and loaded into a query node(also can be loaded into a streaming node, depending on the load balance logic).
Milvus highly depends on ETCD, collection schema, index info, metadata of all segments, runtime status are stored in etcd. Each component will read/write data into etcd constantly. If the etcd is deployed on a low-performance disk, we will see lots of slow request warnings in etcd log. Milvus has many concurrent goroutines, and continually slow etcd requests will break some goroutines and lead to unexpected behavior. Eventually, some components might hit panic logic and quit. We strongly recommend deploying etcd on an NVMe disk.
When a streaming node is restarted, the collection whose shards are on this node will become unsearchable. milvus automatically loads the collection. Then it runs into recovery, waits for the streaming node to connect to the message queue, and syncs the streaming data, and wait other query nodes to load the sealed segments of the collection.
So far as I know, you don't need to care any order while deploying milvus helm, all components are automatically synced and connected. No post steps required, too.
Since you can't provide a full log, it is hard to say what the root cause is. The common reason is a slow disk for etcd,

[Siddeshvr]

@yhmo @xiaofan-luan
Thanks for your input.

I spent some additional time analyzing the operational flow. The issue occurs during a rolling restart of the Milvus instance. Since the StreamingNode acts as a shard leader, when it restarts, a new pod comes up but the Proxy node still holds a reference to the old StreamingNode.

During this window, if a client sends a search/query request to the Proxy, it attempts to route the request to the old StreamingNode (which was previously the shard leader but is no longer available). As a result, the Proxy throws the following error:
MilvusClientException: no available shard leaders: channel not available

In essence, the Proxy node fails to synchronize with the available backend nodes (StreamingNode, DataNodes, QueryNode, MixCoord) after the rolling restart.

As a workaround, I restarted all the nodes again, after which the system recovered to a stable state.

[yhmo]

Normally, before you manually stop a streaming node, you need to start a new streaming node so that the old SN can transfer data to the new SN. Once the old SN finishes data transfer, it will gracefully stop. And the proxy will redirect to the new SN. Anyway, manual operation is not recommended to maintain a milvus cluster. The doc already mentioned how to upgrade a milvus cluster by operator or helm:
https://milvus.io/docs/upgrade_milvus_cluster-operator.md
https://milvus.io/docs/upgrade_milvus_cluster-helm.md

[Siddeshvr]

@yhmo
I am currently running Milvus v2.6.12 using Helm. Whenever I need to restart Milvus, I simply execute the following command:
helm upgrade --install ..............

I am not enforcing any specific restart order and rely entirely on Milvus to manage the sequence of node restarts.

At present, Milvus is deployed across more than 30 environments, but this issue has only been observed in 2 of them during restarts.

I am using a rolling upgrade strategy, where new nodes are brought up before old ones are terminated. For example, with a minimum of 2 replicas per node, the rolling upgrade process spins up 1 new replica and then brings down 1 old replica. This continues until both replicas are replaced with new ones across all components.

Do I need to configure anything specific in Helm for Milvus upgrades to handle this scenario better?