eBPF on top of self build wireguard not working

[joniw]

Hello,

i try to use Calico in a two-node cluster. One node is located behind a NAT-Gateway at home, the other one is a vps with a public ip. The idea is to run the pods on the "internal" node at home and use the public ip as ingress and in a later step as egress. To preserve the source ip, i want to use the ebpf mode.

The nodes are directly connected with a wireguard vpn, the needed ports are forwarded by the NAT-Gateway, including source NAT for outgoing wireguard packages. The nodes can ping each other wireguard interfaces and the cluster is setup in ebpf mode and working as expected. The kubernetes nodes are bound to the wireguard-ips.

I tried purelb for assigning the external ips. As the vps only has one interface, the external ipv4 address is assigned on startup, purelb picks it up as expected.

When starting a whoami pod on the internal node and running curl from another external vps, the packets are packaged into the VXLan and appear at the internal node but are not forwarded to the pod:

• HOST.IP.EXTERNAL.NODE = IP of the node where curl ist running
• EXT.ERNAL.IPV4.VPSNODE= external ip of the vps node)

tcpdump -ni any "host HOST.IP.EXTERNAL.NODE or host $(kubectl get -l app.kubernetes.io/component=whoami pod -o jsonpath="{ .items[0].status.podIPs[0].ip }") or host $(kubectl get -l app.kubernetes.io/component=whoami service -o jsonpath="{ .items[0].spec.clusterIPs[0] }") or (host EXT.ERNAL.IPV4.VPSNODE and port 80)" or port 4789
tcpdump: WARNING: any: That device doesn't support promiscuous mode
(Promiscuous mode not supported on the "any" device)
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
11:18:35.551780 wg0   In  IP 192.168.201.30.41548 > 192.168.201.10.4789: VXLAN, flags [I] (0x08), vni 13242816
IP HOST.IP.EXTERNAL.NODE.41500 > EXT.ERNAL.IPV4.VPSNODE.80: Flags [S], seq 3214982617, win 64240, options [mss 1460,sackOK,TS val 2781779746 ecr 0,nop,wscale 7], length 0
11:18:36.568665 wg0   In  IP 192.168.201.30.41548 > 192.168.201.10.4789: VXLAN, flags [I] (0x08), vni 13242816
IP HOST.IP.EXTERNAL.NODE.41500 > EXT.ERNAL.IPV4.VPSNODE.80: Flags [S], seq 3214982617, win 64240, options [mss 1460,sackOK,TS val 2781780764 ecr 0,nop,wscale 7], length 0
11:18:37.618464 wg0   In  IP 192.168.201.30.41548 > 192.168.201.10.4789: VXLAN, flags [I] (0x08), vni 13242816
IP HOST.IP.EXTERNAL.NODE.41500 > EXT.ERNAL.IPV4.VPSNODE.80: Flags [S], seq 3214982617, win 64240, options [mss 1460,sackOK,TS val 2781781814 ecr 0,nop,wscale 7], length 0
11:18:38.658393 wg0   In  IP 192.168.201.30.41548 > 192.168.201.10.4789: VXLAN, flags [I] (0x08), vni 13242816
IP HOST.IP.EXTERNAL.NODE.41500 > EXT.ERNAL.IPV4.VPSNODE.80: Flags [S], seq 3214982617, win 64240, options [mss 1460,sackOK,TS val 2781782854 ecr 0,nop,wscale 7], length 0

When using IPv6, the package seems to be truncated:

tcpdump -ni any host fd05:2046:b984:a310::10 or host $(kubectl get -l app.kubernetes.io/component=whoami pod -o jsonpath="{ .items[0].status.podIPs[1].ip }") or host $(kubectl get -l app.kubernetes.io/component=whoami service -o jsonpath="{ .items[0].spec.clusterIPs[1] }") or host EXTERNAL.IPV6.CURLNODE or host EXTERNAL.IPV6.VPSNODE or port 4789
tcpdump: WARNING: any: That device doesn't support promiscuous mode
(Promiscuous mode not supported on the "any" device)
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
11:19:44.293810 wg0   In  IP6 fd05:2046:b984:a311::30.45770 > fd05:2046:b984:a311::10.4789: VXLAN, flags [I] (0x08), vni 13242816
[length 74 < caplen 94] (invalid)
11:19:46.408892 wg0   In  IP6 fd05:2046:b984:a311::30.45770 > fd05:2046:b984:a311::10.4789: VXLAN, flags [I] (0x08), vni 13242816
[length 74 < caplen 94] (invalid)
11:19:48.499550 wg0   In  IP6 fd05:2046:b984:a311::30.45770 > fd05:2046:b984:a311::10.4789: VXLAN, flags [I] (0x08), vni 13242816
[length 74 < caplen 94] (invalid)

Because of the behavior with ipv6 packages i set the mtu to 1350 but this also did not work.

Without ebpf mode, everything is working as expected. It also works when i move the whoami pod to the vps node. Also using curl on the cluster nodes within the cluster, the connection is successful.

For me it looks like some bpf program is not processing the packages as it should. I also tried to end the wireguard connection on the NAT-Gateway, then i have to use VXLan for encapsulation. Then the packets arrive at the pod but the answer packages are not picked up on the wireguard interface of the vps node.

I also added the wireguard interface to bpfDataIfacePattern but then the wireguard connection was not working at all.

I am thankful for hints and advices to make this setup work.

With regards

Joniw

Expected Behavior

Using curl on the external ip in ebpf mode should lead to a working connection and return the correct source ip.

Current Behavior

The connect package is received by the vps node and forwarded through the VXLAN-Tunnel. Then the package is not getting forwarded to the pod and the connect fails.

Possible Solution

I have no idea, perhaps there is an error in the configuration?

Steps to Reproduce (for bugs)

1. Setup two cluster nodes with calico in ebpf node and connect them with a wireguard tunnel. Use the tunnel ips as the kubernetes interfaces.
2. Add an "external" ip via metallb or purelb on node1 on the ethX or other external interface
3. Install a whoami container on node2
4. Try to curl the whoami-page from a third node or client

Context

Run pods on internal nodes behind a NAT and use the public ip of the vps as ingress (and egress).

Your Environment

• Calico version: v3.30.5 (via operator, v3.31 is not working because of [v3.31] ebpf - Split policy jump map #11488)
• Calico dataplane: bpf
• Orchestrator version (e.g. kubernetes, openshift, etc.): k3s version v1.34.1+k3s1 (24fc436e)
• Operating System and version: AlpineLinux (Kairos 3.6.0)

k3s configuration:

node-ip: 192.168.201.10,fd05:2046:b984:a311::10
disable:
  - servicelb
  - traefik
  - local-storage
cluster-cidr: 10.42.0.0/16,fd05:2046:b984:a320::0/60
service-cidr: 10.43.0.0/16,fd05:2046:b984:a330::0/108
flannel-backend: none
disable-kube-proxy: true
disable-network-policy: true
cluster-init: true

tigera-operator.yaml:

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: tigera-operator
  namespace: kube-system
spec:
  # for values see https://docs.k3s.io/helm
  repo: https://docs.tigera.io/calico/charts
  chart: tigera-operator
  version: v3.30.5
  targetNamespace: tigera-operator
  createNamespace: true
  bootstrap: True
  valuesContent: |-
    kubernetesServiceEndpoint:
      host: 192.168.201.10
      port: 6443
    goldmane:
      enabled: false
    whisker:
      enabled: false
    defaultFelixConfiguration:
      enabled: true
      bpfL3IfacePattern: ^wg0$
    installation:
      controlPlaneReplicas: 1
      calicoNetwork:
        mtu: 1350
        bgp: Enabled
        linuxDataplane: BPF
        nodeAddressAutodetectionV4:
          kubernetes: NodeInternalIP
        nodeAddressAutodetectionV6:
          kubernetes: NodeInternalIP
        ipPools:
          - blockSize: 26
            cidr: 10.42.0.0/16
            encapsulation: None
            natOutgoing: Enabled
            nodeSelector: all()
          - blockSize: 120
            cidr: fd05:2046:b984:a320::0/60
            encapsulation: None
            natOutgoing: Enabled
            nodeSelector: all()