weave-net pods restarting on kubernetes

[Cryptophobia]

This has been happening for a long time and we have no idea what is going on. weave-net pods are restarting with the following error: Warning SandboxChanged Pod sandbox has changed, containers will be killed and recreated.

Below is how the weave-net restarts look in kubectl:

weave-net-44z47                                                       2/2       Running             3          4d
weave-net-4gpkz                                                       1/2       CrashLoopBackOff    940        4d
weave-net-5rjbs                                                       2/2       Running             0          4d
weave-net-5wbsb                                                       2/2       Running             0          4d
weave-net-84rbx                                                       0/2       ContainerCreating   0          53s
weave-net-bvxgz                                                       2/2       Running             1          4d
weave-net-cgdkc                                                       2/2       Running             10         4d
weave-net-dbkfm                                                       2/2       Running             1          4d
weave-net-h8rkv                                                       2/2       Running             0          4d
weave-net-lwnfv                                                       2/2       Running             0          4d
weave-net-lxxmp                                                       2/2       Running             0          4d
weave-net-mbsmp                                                       0/2       CrashLoopBackOff    983        4d

What you expected to happen?

No restarts on weave-net pods. We are seeing over 500 restarts on some weave-net pods.

What happened?

Warning SandboxChanged Pod sandbox has changed, containers will be killed and recreated.

How to reproduce it?

No idea.

Anything else we need to know?

We use kops version 1.8.1 to configure our prod and staging clusters on AWS.

Versions:

weave-net: 2.2.1
docker version: 17.09

$ weave version
$ docker version
$ uname -a
4.9.0-6-amd64
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T12:22:21Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.10", GitCommit:"044cd262c40234014f01b40ed7b9d09adbafe9b1", GitTreeState:"clean", BuildDate:"2018-03-19T17:44:09Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Logs:

$ docker logs weave

or, if using Kubernetes:

$ kubectl logs -n kube-system <weave-net-pod> weave

Network:

$ ip route
$ ip -4 -o addr
$ sudo iptables-save

[bboreham]

Can you get the kubelet log too, for one of the hosts where it is restarting?

[Cryptophobia]

The file is huge but here is a sample of the weave errors. Lots of socket errors and "CNI failed to retrieve connection namespace path."

kubelet.log

Would you like me to upload the whole journalctl output somewhere?

[bboreham]

The errors you show are caused by the Weave Net container being restarted, in other words that is a symptom not the cause. We need to know more about why kubelet has restarted the Weave Net container.

Generally it's a waste of time to try to cut the file down to just what is needed, because neither of us knows which bits are relevant.

[bboreham]

Doing some searching, it seems this can be caused by the docker daemon restarting. Please could you have a look at its logs too?

If there's nothing in the docker logs, check that the docker-healthcheck script installed by kops is not killing it. I've seen that before.

[sstarcher]

I'm seeing this same issue on kubernetes 1.9.3, weave 2.2.0, kops 1.9.0-beta-2. When it happens again I'll try and capture logging information. It has happened twice in 2 weeks.

[bboreham]

To be clear, "sandbox has changed" is Kubelet reacting to what it sees as a change, and "sandbox" is how it refers to the underlying container. For example, Docker restarting, causing all containers to restart, would be one way to cause this. If you have an issue on your node(s) that causes Docker to restart, then everything will be impacted, including Weave Net.

[Cryptophobia]

So we knew about the issue that dockerd was restarting but not sure why that was happening. We made slight changes last night. One of the changes was clearly define memory and cpu limits for most pods in our k8s cluster. The other change was to provision more IOPS to the nodes and masters. We have since seen significantly less restarts, almost none.

[bboreham]

Good to hear - do you think it was the healthcheck job that was killing it?

[Cryptophobia]

@bboreham Yes, exactly! The docker-healthcheck script was restarting docker since this line if timeout 10 docker ps > /dev/null must have been taking longer than the allowed timeout because of low IOPS on the nodes...

[Cryptophobia]

Hope this helps other people out there. I will open an issue with kops. Maybe there should be another healthcheck service running benchmarking iops on the server or the docker-healthcheck script can be refactored as it turns into a single point of failure when the servers are under high load and with low IOPS?

[sstarcher]

I would be surprised if that was my issue due to the extremely low load on my nodes, but I can't verify it until the issues happens again.

[Cryptophobia]

@sstarcher : I discussed with my team again to check if there was anything we changed that was not related to IOPS. The only other thing we noticed is that it could be resource limits based. We noticed when we did not have strict resource limits on some services they would hog cpu and memory from the nodes. In our case it was a logging service that was not buffering the logs correctly and therefore requesting lots of cpu and memory to push them out.

Is it possible you had some service running on your cluster that was using lots of cpu or memory at intermittent times? That was the only other problem we noticed.