panic: etcdserver: leader changed

[Somebi]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via security@etcd.io.
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

Including full log here from standalone milvus docker, which fails due to etcd:
https://gist.github.com/Somebi/9d4ac6ae975eae1c32bc4255969eb932

What did you expect to happen?

expected not to have panic

How can we reproduce it (as minimally and precisely as possible)?

no idea

Anything else we need to know?

No response

Etcd version (please run commands below)

Details

version 3.5.23

Etcd configuration (command line flags or environment variables)

Details

paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

Details

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

[Sayan-995]

The embedded etcd starts simultaneously with Milvus components, which is most probably causing this race. Try running a dedicated etcd container/service that's fully healthy before Milvus starts , It should eliminate this problem

[Somebi]

So the problem is, that milvus is trying to use etcd while it is not ready?

[Somebi]

So they are not handling raft leader changing case properly and fail early, without waiting for raft to elect new leader? This, causing container to fall into the restart loop.

[Sayan-995]

Yes , rejecting in-flight requests during a leader election is expected Raft behavior. The problem is that Milvus treats this transient error as fatal .

[mehrdadbn9]

Hi, I'd like to work on this leader change panic issue. The stack trace suggests a race condition during leader transitions. Could a maintainer please assign it to me? Thank you!