[HELP] MetalLB (v0.11.0) and K3D timeout/no route to host

[jessequinn]

Question / Where do you need Help?

MetalLB v0.11.0 is installed, K3D is used to create a 1 server 2 agent cluster.

More details can be seen in the template:

apiVersion: k3d.io/v1alpha2
kind: Simple
name: {{ .ClusterName }}
servers: 1
agents: 2
image: rancher/k3s:v1.21.2-k3s1
registries:
  create: false
  use:
    - k3d-registry.{{ .Domain }}:{{ .Port }}
  config: |
    mirrors:
      "us.gcr.io":
        endpoint:
          - http://k3d-registry.{{ .Domain }}:{{ .Port }}
      "docker.io":
        endpoint:
          - http://k3d-registry.{{ .Domain }}:{{ .Port }}
      "quay.io":
        endpoint:
          - http://k3d-registry.{{ .Domain }}:{{ .Port }}
options:
  k3d:
    wait: true
    timeout: "200s"
    disableLoadbalancer: false
    disableImageVolume: false
    disableRollback: false
    disableHostIPInjection: false
  k3s:
    extraServerArgs:
      - --no-deploy=traefik
      - --disable=servicelb
    extraAgentArgs:
      - --kubelet-arg=eviction-hard=imagefs.available<1%,nodefs.available<1%
      - --kubelet-arg=eviction-minimum-reclaim=imagefs.available=1%,nodefs.available=1%
  kubeconfig:
    updateDefaultKubeconfig: true
    switchCurrentContext: true

Our problem is the following:

After a specific number of IPs created via MetalLB, roughly 4-5, any subsequent IPs created result in services being timed out / no route to host. What i have discovered so far, only one node, of the three, works with MetalLB. The other two nodes that have any MetalLB IPs routing to them result in the aforementioned error.

Does anyone have ideas how we can tackle this issue? My kubernetes environment has not really changed, yet, this problem is now just appearing when creating new environments.

As a side note, we added tcpdump sidecars to deployments as simple as nginx and exposing port 80 to create an IP and the container is not receiving any requests. However, the speaker for MetalLB appears to be forwarding but not receiving anything back.

Any help would be greatly appreciated.

Scope of your Question

k3d version v4.4.7
k3s version v1.21.2-k3s1 (default)

metallb v0.11.0

Update

(moved from separate comment by @iwilltry42)

In addition, I getting this error in one of the speaker pods for MetalLB

│ {"caller":"state.go:459","component":"Memberlist","level":"info","msg":"memberlist: Suspect k3d-cluster-test-agent-1 has failed, no acks received","ts":"2021-12-17T17:56:23.900003962Z"}                                            │
│ {"caller":"state.go:459","component":"Memberlist","level":"info","msg":"memberlist: Suspect k3d-cluster-test-agent-0 has failed, no acks received","ts":"2021-12-17T17:56:25.540462539Z"}   

entering the container node and running arping

/ # arping -I eth0 172.19.100.18
ARPING 172.19.100.18 from 172.19.0.3 eth0

while the other two

/ # arping -I eth0 172.19.100.18
ARPING 172.19.100.18 from 172.19.0.5 eth0
Unicast reply from 172.19.100.18 [02:42:ac:13:00:03] 0.015ms
Unicast reply from 172.19.100.18 [02:42:ac:13:00:03] 0.447ms
Unicast reply from 172.19.100.18 [02:42:ac:13:00:03] 0.181ms

/ # arping -I eth0 172.19.100.18
ARPING 172.19.100.18 from 172.19.0.4 eth0
Unicast reply from 172.19.100.18 [02:42:ac:13:00:03] 0.303ms
Unicast reply from 172.19.100.18 [02:42:ac:13:00:03] 0.252ms
Unicast reply from 172.19.100.18 [02:42:ac:13:00:03] 0.174ms
Unicast reply from 172.19.100.18 [02:42:ac:13:00:03] 0.138ms
^CSent 4 probe(s) (1 broadcast(s))
Received 4 response(s) (0 request(s), 0 broadcast(s))

[iwilltry42]

Hi @jessequinn , thanks for starting this discussion!
Unfortunately, I have no experience with MetalLB and don't know how it works internally.
I moved your issue to a discussion and posted it in the public #k3d slack channel in https://rancher-users.slack.com to get more eyes on it.

only one node, of the three, works with MetalLB

Which node is it? Is there a pattern (e.g. always the server node)?
Judging from your update, it seems like the failing speaker is server-0 while the agents node don't have this problem? 🤔

EDIT:
As far as I can see, all (or at least all I've seen) tutorials on k3d with metallb use <=v0.10.2 and it seems like quite a lot changed in v0.11.0 with regards to networking 🤔

[jessequinn]

Originally i was using 0.10.2 and it worked then not. We are using Yocto and I fear something within Yocto is broken. Do you know what iptables modules are required? I did catch an error about Statistics not found.

[jessequinn]

on another note, I ran the same setup on ubuntu. no problem. so definitely something within Yocto I think.

[jessequinn]

time="2021-12-17T17:34:27.739166672Z" level=info msg="Running kube-proxy --cluster-cidr=10.42.0.0/16 --conntrack-max-per-core=0 --conntrack-tcp-timeout-close-wait=0s --conntrack-tcp-timeout-established=0s --healthz-bind-address=127.0.0.1 --hostname-override=k3d-cluster-test-agent-0 --kubeconfig=/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig --proxy-mode=iptables"
I1217 17:34:32.943708       6 kubelet_network_linux.go:56] "Initialized protocol iptables rules." protocol=IPv4
I1217 17:34:32.981486       6 kubelet_network_linux.go:56] "Initialized protocol iptables rules." protocol=IPv6
I1217 17:34:35.537709       6 server_others.go:213] Using iptables Proxier.
I1217 17:34:35.537735       6 server_others.go:220] creating dualStackProxier for iptables.
I1217 17:34:38.850807       6 iptables.go:148] Some iptables rules are missing; deleting and recreating rules
I1217 17:34:38.851133       6 iptables.go:172] Deleting iptables rule: -s 10.42.0.0/16 -d 10.42.0.0/16 -j RETURN
I1217 17:34:38.851879       6 iptables.go:148] Some iptables rules are missing; deleting and recreating rules
I1217 17:34:38.851903       6 iptables.go:172] Deleting iptables rule: -s 10.42.0.0/16 -j ACCEPT
I1217 17:34:38.852188       6 iptables.go:172] Deleting iptables rule: -s 10.42.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully
I1217 17:34:38.853134       6 iptables.go:172] Deleting iptables rule: -d 10.42.0.0/16 -j ACCEPT
I1217 17:34:38.853677       6 iptables.go:172] Deleting iptables rule: ! -s 10.42.0.0/16 -d 10.42.1.0/24 -j RETURN
I1217 17:34:38.853993       6 iptables.go:160] Adding iptables rule: -s 10.42.0.0/16 -j ACCEPT
I1217 17:34:38.854823       6 iptables.go:172] Deleting iptables rule: ! -s 10.42.0.0/16 -d 10.42.0.0/16 -j MASQUERADE --random-fully
I1217 17:34:38.856143       6 iptables.go:160] Adding iptables rule: -d 10.42.0.0/16 -j ACCEPT
I1217 17:34:38.856681       6 iptables.go:160] Adding iptables rule: -s 10.42.0.0/16 -d 10.42.0.0/16 -j RETURN
I1217 17:34:38.858748       6 iptables.go:160] Adding iptables rule: -s 10.42.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully
I1217 17:34:38.860720       6 iptables.go:160] Adding iptables rule: ! -s 10.42.0.0/16 -d 10.42.1.0/24 -j RETURN
I1217 17:34:38.862418       6 iptables.go:160] Adding iptables rule: ! -s 10.42.0.0/16 -d 10.42.0.0/16 -j MASQUERADE --random-fully
E1217 18:05:34.401847       6 proxier.go:1612] "Failed to execute iptables-restore" err="exit status 2 (iptables-restore v1.8.5 (legacy): Couldn't find match `statistic'\n\nError occurred at line: 177\nTry `iptables-restore -h' or 'iptables-restore --help' for more information.\n)"

[iwilltry42]

I think this is not K3s specific, but coming from kube-proxy (I may be wrong here).
I really don't know which modules would be required, but maybe this issue gives you some hints at least on how to debug it: kubernetes/kubernetes#12459

[jessequinn]

resolved - xt_statistic ext missing solved the problem. thanks.