Etcd communication warnings

[chrisarnott86]

Environmental Info:
K3s Version:
v1.31.5+k3s1

Node(s) CPU architecture, OS, and Version:
Linux master1 6.1.0-31-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.128-1 (2025-02-07) x86_64 GNU/Linux

Cluster Configuration:
3 servers, 1 agent

Describe the bug:
I have both metallb and kube-vip installed in my cluster, with the installation of kube-vip I have started seeing warnings logged from the nodes. I should note that although this error is most likely introduced by installing kube-vip, I'm keen to see if anyone can help work out the root cause, even if the solution lies with kube-vip. I'm struggling to work out what's going on, I have also submitted this as an issue with kube-vip but not had any feedback so far.

On two out of three master nodes I'm seeing errors like this logged every few seconds:

Mar 06 15:17:06 master1 k3s[796]: {"level":"warn","ts":"2025-03-06T15:17:06.641715Z","caller":"embed/config_logging.go:161","msg":"rejected connection on peer endpoint","remote-addr":"X.X.12.57:50750","server-name":"","ip-addresses":["X.X.12.130","127.0.0.1","::1","X.X.12.128","10.43.0.1"],"dns-names":["kine.sock","localhost","master3"],"error":"tls: \"X.X.12.57\" does not match any of DNSNames [\"kine.sock\" \"localhost\" \"master3\"] (lookup kine.sock: no such host)"}

The remote-addr is always the IP of one of the LoadBalancer service in the cluster, and it seems to cycle through a lot of ports for each address. So in the example it's trying to contact X.X.12.57 which isn't one of the nodes but is an address used by a LoadBalancer service.

The three nodes have the addresses:

master1 - X.X.12.124
master2 - X.X.12.116
master3 - X.X.12.128

Kube-vip address for api access: X.X.12.130

Then on the third master I'm seeing messages like:

Mar 06 15:18:16 k3s3 k3s[779]: {"level":"warn","ts":"2025-03-06T15:18:16.137995Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"<node id of 1 other node>","rtt":"5.984007ms","error":"EOF"}

Steps To Reproduce:

• Installed K3s:
• installed metallb
• installed kube-vip (with egress enabled)

Expected behavior:
Communication between nodes continues as normal

Actual behavior:
Errors with communication for the embedded etcd service are being logged

[brandond]

What are you using for the k3s datastore? I see references to kine.sock which would suggest you're using kine, but there are also message from etcd raft which would only be present if you're using embedded etcd.

Did you perhaps configure kube-vip to do something with the etcd peer port? Health-check it, or expose the etcd peer port via the VIP, or something like that? You should not be doing anything with the etcd ports in your kube-vip config.

[chrisarnott86]

Thanks @brandond, I was confused about the messages for kine, as the cluster is just using the default embedded etcd with k3s, I also haven't done anything with kube-vip in relation to etcd so I'm not sure where the kine messages were coming from. This morning the cluster had completely gummed up, none of the masters were working each with etcd related errors. I did a restore from etcd snapshot and once it was all up I hastily removed kube-vip! It seems like kube-vip caused some major problem for etcd but I don't understand how? In terms of the errors have you seen anything before that would result in those kind of issues for etcd?

[brandond]

What is the remote-addr in those log messages? I'd probably try to figure out what is going on that is causing things to start connecting to the peer port when kube-vip is in use. Kube-vip generally works quite well, and I'm not sure why anything would be trying to connect to that port. Only other etcd cluster members should connect to that.

[chrisarnott86]

The remote-addr in the logs was variable, but it was always the IP of one of the LoadBalancer services defined in the cluster.

[chrisarnott86]

I've done more testing which suggests very strongly that the issue is solely with kube-vip (see kube-vip/kube-vip#1061). I'll consider this question closed/answered.