Netbird operator does not work when cilium+kubeproxy replacement is active

[bornav]

When installing netbird operator on a cluster that does not have default kube scheduler, netbird operator deploys routers and resources to the netbird-api plane successfuly, assigns network and matches the resource to the domain, but after manually adding the policy when trying to access the resource it fails with error ERR_CONNECTION_TIMED_OUT

for testing i tried it on a selhosted instance and the publicly available one, both result in same behavior

i can easily reproduce this by deploying it in a kind cluster, i provided 3 scripts for deploying it normally with default cni, with cilium without kube proxy replacement, and cilium with kube proxy replacement

in this repo https://github.com/bornav/cluster-playground/tree/main/netbird-operator you can find 3 deploy scripts

deploy.sh (cilium cni, with kube proxy replaced) # broken
deploy-normal.sh (normal cni) # working
deploy-cilium-w-proxy.sh (cilium cni, with default kube proxy) # working

just running the script locally should give a reproduced example, assuming netbird/secret.yaml is populated

cleanup would be just a matter of running cleanup.sh

i dont see any errors on operator/networkrouters that would be related to this

policy that is manually created

resource assigned but not actually accessible

[knordum]

Can confirm we also experience the same issue when using Cilium with kubeProxyReplacement: true.

In our case the NetBird operator reconciles successfully and the network router becomes Ready, but traffic through the NetBird-routed resource does not seem to go through.

[phillebaba]

My best guess without having looked into this yet is that the iptables setup by the netbird client does not play well with the kube proxy replacement that circumvents the need for iptables/nftables/ipvs. I have experienced other issues with running Cilium without kube proxy in the past as other projects assume that the networking stack functions in a specific manner.

I have some other work that needs to be completed first before I can dig into figuring out where things are going wrong. I can assume that this will take some work to trace. If anyone else has does end up trying to figure out where packets are dropped please do share what you find!

[bornav]

@phillebaba i did a quick pcap of it and shat it into llm and here is what it came up with

The packets are being dropped between the DNAT step and the pod delivery. With Cilium replacing kube-proxy:

• Standard iptables DNAT/SNAT rules for Service port forwarding exist (that's why we see the DNAT)
• But Cilium uses eBPF-based L7/L4 proxying instead of iptables for actual packet forwarding
• Cilium's eBPF programs likely don't recognize the DNAT'd UDP packets as belonging to a Service endpoint, so they drop them
• TCP traffic to kube-apiserver works because it goes through Cilium's own proxy handling on lo

not sure how much i would trust this but had no time to manually investigate the pcap, but ill provide it here

dumpfile.tar.gz

i have attached the pcap of it, this is from the example above that is broken if it helps

[Kariton]

i do run cilium with kubeProxyReplacement: true and native routing with bpf.datapathMode: netkit + bpf.masquerade: true successfully.
ive setup NetworkRouter and a NetworkResource that points to an envoy-gateway service.

works as expected in my PoC.

then ive created some netbird reverse-proxy entries that point to that netbird resource and pass the host header down to that resource.

also "internal" (other peers; mesh), via netbird dns, where ive created CNAME records for my NetworkRouter domain records that point to the service.

• NetworkRouter: cluster.example.com (operator managed)
  • envoy.envoy-external.cluster.example.com -> netbird resource
• new zone: example.com (manually created)
  • cname X.example.com -> envoy.envoy-external.cluster.example.com

only downside with other peers is the router nat which prevents me from keeping the real ips from clients. (100.64.0.0/10 subnet ip)
instead ill get the NAT address from my cluster internal netbird router container. (as expected; as documented)
just need to figure out how to setup SidecarProfile (which crashes due to user permissions and ro-fs etc.)
or another way to make that work as expected.

most relevant cilium values in my deployment:

    k8sServiceHost: 127.0.0.1
    k8sServicePort: 6443

    kubeProxyReplacement: true
    kubeProxyReplacementHealthzBindAddr: 0.0.0.0:10256

    ipam:
      mode: kubernetes

    ipv6:
      enabled: true

    devices: bond+

    # Cilium routing for inter-nodes pod traffic
    routingMode: native
    autoDirectNodeRoutes: true
    directRoutingSkipUnreachable: true
    localRedirectPolicies:
      enabled: true
    ipv4NativeRoutingCIDR: 10.244.0.0/16
    ipv6NativeRoutingCIDR: fd00:10:244::/64
    loadBalancer:
      acceleration: best-effort
      algorithm: maglev
      mode: hybrid

    k8s:
      requireIPv4PodCIDR: true
      requireIPv6PodCIDR: true

    bgpControlPlane:
      enabled: true
      routerIDAllocation:
        mode: ip-pool
        ipPool: 10.244.0.0/24

    ciliumEndpointSlice:
      enabled: true

    # expected to become default - soonTM; https://github.com/cilium/cilium/issues/14955
    endpointRoutes:
      enabled: true

    bandwidthManager:
      enabled: true
      bbr: true

    bpf:
      datapathMode: netkit
      masquerade: true

    enableIPv6BIGTCP: true
    enableIPv4BIGTCP: true
    bpfClockProbe: true

    socketLB:
      enabled: true
      hostNamespaceOnly: true

[bornav]

@Kariton after doing A B testing only thing that made a diff was

socketLB:
  enabled: true
  hostNamespaceOnly: true

[bornav]

@phillebaba considering it's fixable via cilium value change i would consider this as fixed

[phillebaba]

I am going to reopen the issue because I think that we should document the required configuration before closing.

[christianhuth]

For my environment setting

socketLB:
  enabled: true
  hostNamespaceOnly: true

didn't fix the problem. Still looking for a solution. See this issue as well: #284

[phillebaba]

@christianhuth could we keep the discussion here if your issue is related to Cilium kube-proxy replacement. Makes it easier for me to track context and to keep information concentrated.

If not we can keep looking at things in #284.

[christianhuth]

@phillebaba: Sure. But I don't think the socketLB nor the kubeProxyReplacement is the real problem. I have a local kind cluster running with kubeProxyReplacement and without socketLB and netbird works just fine. I can share the code for that environment if it helps you.

[bornav]

@christianhuth in my case i had local rke2 cluster, could it since it's running under docker for you?
Id still say it's an issue, but to try to narrow it down