stop/start microk8s leaks IP addresses on flannel

[d-shehu]

Summary

When rebooting or otherwise making changes to microk8s that require it to be stopped and restarted, I noticed that flannel periodically "leaks" IP addresses. More IPs are used until the range is exhausted and then microk8s is no longer able to run pods.

This happens even if the cluster is otherwise static, no deployments, no new nodes, etc.

1. HA is disabled
2. Single node

sudo ls /var/lib/cni/networks/microk8s-flannel-network
10.1.42.10  10.1.42.14  10.1.42.18  10.1.42.22  10.1.42.26  10.1.42.3   10.1.42.33  10.1.42.37  10.1.42.41  10.1.42.45  10.1.42.49  10.1.42.52  10.1.42.56  10.1.42.8
10.1.42.11  10.1.42.15  10.1.42.19  10.1.42.23  10.1.42.27  10.1.42.30  10.1.42.34  10.1.42.38  10.1.42.42  10.1.42.46  10.1.42.5   10.1.42.53  10.1.42.57  10.1.42.9
10.1.42.12  10.1.42.16  10.1.42.2   10.1.42.24  10.1.42.28  10.1.42.31  10.1.42.35  10.1.42.4   10.1.42.43  10.1.42.47  10.1.42.50  10.1.42.54  10.1.42.6   last_reserved_ip.0
10.1.42.13  10.1.42.17  10.1.42.21  10.1.42.25  10.1.42.29  10.1.42.32  10.1.42.36  10.1.42.40  10.1.42.44  10.1.42.48  10.1.42.51  10.1.42.55  10.1.42.7   lock

What Should Happen Instead?

Microk8s should gracefully stop/start without having flannel network failing periodically.

Reproduction Steps

1. microk8s stop
2. Make changes for example:

 /var/lib/cni/networks/microk8s-flannel-network

3. microk8s start

or

1. Reboot
2. Rinse and repeat

Introspection Report

N/A

Microk8s inspect was run and it didn't flag the network issue nor otherwise resolve the problem.

Can you suggest a fix?

The "hacky" workaround is to stop microk8s and remove the dangling IPs

sudo rm -rf /var/lib/cni/networks/microk8s-flannel-network

Are you interested in contributing with a fix?

[christoph-h]

I see the same issue with microk8s after disabling HA and thus switching to flannel. I'm on MicroK8s v1.33.5 revision 8507.

sudo ls /var/lib/cni/networks/microk8s-flannel-network
10.1.16.10   10.1.16.114  10.1.16.129  10.1.16.143  10.1.16.158  10.1.16.172  10.1.16.187  10.1.16.200  10.1.16.215  10.1.16.23   10.1.16.244  10.1.16.3   10.1.16.44  10.1.16.59  10.1.16.73  10.1.16.88
10.1.16.100  10.1.16.115  10.1.16.13   10.1.16.144  10.1.16.159  10.1.16.173  10.1.16.188  10.1.16.201  10.1.16.216  10.1.16.230  10.1.16.245  10.1.16.30  10.1.16.45  10.1.16.6   10.1.16.74  10.1.16.89
10.1.16.101  10.1.16.116  10.1.16.130  10.1.16.145  10.1.16.16   10.1.16.174  10.1.16.189  10.1.16.202  10.1.16.217  10.1.16.231  10.1.16.246  10.1.16.31  10.1.16.46  10.1.16.60  10.1.16.75  10.1.16.9
10.1.16.102  10.1.16.117  10.1.16.131  10.1.16.146  10.1.16.160  10.1.16.175  10.1.16.19   10.1.16.203  10.1.16.218  10.1.16.232  10.1.16.247  10.1.16.32  10.1.16.47  10.1.16.61  10.1.16.76  10.1.16.90
10.1.16.103  10.1.16.118  10.1.16.132  10.1.16.147  10.1.16.161  10.1.16.176  10.1.16.190  10.1.16.204  10.1.16.219  10.1.16.233  10.1.16.248  10.1.16.33  10.1.16.48  10.1.16.62  10.1.16.77  10.1.16.91
10.1.16.104  10.1.16.119  10.1.16.133  10.1.16.148  10.1.16.162  10.1.16.177  10.1.16.191  10.1.16.205  10.1.16.22   10.1.16.234  10.1.16.249  10.1.16.34  10.1.16.49  10.1.16.63  10.1.16.78  10.1.16.92
10.1.16.105  10.1.16.12   10.1.16.134  10.1.16.149  10.1.16.163  10.1.16.178  10.1.16.192  10.1.16.206  10.1.16.220  10.1.16.235  10.1.16.25   10.1.16.35  10.1.16.5   10.1.16.64  10.1.16.79  10.1.16.93
10.1.16.106  10.1.16.120  10.1.16.135  10.1.16.15   10.1.16.164  10.1.16.179  10.1.16.193  10.1.16.207  10.1.16.221  10.1.16.236  10.1.16.250  10.1.16.36  10.1.16.50  10.1.16.65  10.1.16.8   10.1.16.94
10.1.16.107  10.1.16.121  10.1.16.136  10.1.16.150  10.1.16.165  10.1.16.18   10.1.16.194  10.1.16.208  10.1.16.222  10.1.16.237  10.1.16.251  10.1.16.37  10.1.16.51  10.1.16.66  10.1.16.80  10.1.16.95
10.1.16.108  10.1.16.122  10.1.16.137  10.1.16.151  10.1.16.166  10.1.16.180  10.1.16.195  10.1.16.209  10.1.16.223  10.1.16.238  10.1.16.252  10.1.16.38  10.1.16.52  10.1.16.67  10.1.16.81  10.1.16.96
10.1.16.109  10.1.16.123  10.1.16.138  10.1.16.152  10.1.16.167  10.1.16.181  10.1.16.196  10.1.16.21   10.1.16.224  10.1.16.239  10.1.16.253  10.1.16.39  10.1.16.53  10.1.16.68  10.1.16.82  10.1.16.97
10.1.16.11   10.1.16.124  10.1.16.139  10.1.16.153  10.1.16.168  10.1.16.182  10.1.16.197  10.1.16.210  10.1.16.225  10.1.16.24   10.1.16.254  10.1.16.4   10.1.16.54  10.1.16.69  10.1.16.83  10.1.16.98
10.1.16.110  10.1.16.125  10.1.16.14   10.1.16.154  10.1.16.169  10.1.16.183  10.1.16.198  10.1.16.211  10.1.16.226  10.1.16.240  10.1.16.26   10.1.16.40  10.1.16.55  10.1.16.7   10.1.16.84  10.1.16.99
10.1.16.111  10.1.16.126  10.1.16.140  10.1.16.155  10.1.16.17   10.1.16.184  10.1.16.199  10.1.16.212  10.1.16.227  10.1.16.241  10.1.16.27   10.1.16.41  10.1.16.56  10.1.16.70  10.1.16.85  last_reserved_ip.0
10.1.16.112  10.1.16.127  10.1.16.141  10.1.16.156  10.1.16.170  10.1.16.185  10.1.16.2    10.1.16.213  10.1.16.228  10.1.16.242  10.1.16.28   10.1.16.42  10.1.16.57  10.1.16.71  10.1.16.86  lock
10.1.16.113  10.1.16.128  10.1.16.142  10.1.16.157  10.1.16.171  10.1.16.186  10.1.16.20   10.1.16.214  10.1.16.229  10.1.16.243  10.1.16.29   10.1.16.43  10.1.16.58  10.1.16.72  10.1.16.87

The suggested workaround works for me, but all IPs are leaked within about 2 days and the cluster becomes unusable again.

[ASk1]

is this related to ?
flannel-io/flannel#2264

[christoph-h]

@ASk1 Thanks for the hint! Yes, this reads very similar, since we also see the IPs not cleaned up and IPs start accumulating in /var/lib/cni/networks/microk8s-flannel-network (not that the referenced ticket states a bit different path: /var/lib/cni/networks/cbr0)

[d-shehu]

Thanks for the clarification. I'll follow up on the flannel thread.

Btw I wasn't explicitly looking to use flannel. I briefly considered switching to Calico. I switched CNI when I had to disable HA due to corruptions / dqlite losing quorom at random under idle load.

I didn't use Calico because I've found its config to be complex in vanilla k8s.

[ASk1]

I ran into issues with calico on a wsl2 microk8s node and also switched to flannel by disabling HA.