Network performance degradation after RKE2 installation on Debian 12

[ram-suradkar]

I'm seeing peculiar issue after installation of RKE2 on Debian bookworm, network speed between node get halved as soon as rke2 is installed.

Setup Details :

• OS: Debian 12 bookworm
• RKE2
  • Followed for installation https://docs.rke2.io/install/quickstart
  • 1 Control plane, 1 Worker node setup

# kubectl version
Client Version: v1.31.5+rke2r1
Kustomize Version: v5.4.2
Server Version: v1.31.5+rke2r1

• Tested using iperf3, before RKE2 installation -

Server :

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  3.00 GBytes  2.58 Gbits/sec                  receiver
[  8]   0.00-10.00  sec  13.7 GBytes  11.8 Gbits/sec                  receiver
[ 10]   0.00-10.00  sec  2.73 GBytes  2.35 Gbits/sec                  receiver
[ 12]   0.00-10.00  sec  2.98 GBytes  2.56 Gbits/sec                  receiver
[ 14]   0.00-10.00  sec  13.7 GBytes  11.8 Gbits/sec                  receiver
[ 16]   0.00-10.00  sec  2.78 GBytes  2.39 Gbits/sec                  receiver
[ 18]   0.00-10.00  sec  2.88 GBytes  2.47 Gbits/sec                  receiver
[ 20]   0.00-10.00  sec  13.7 GBytes  11.8 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  55.5 GBytes  47.6 Gbits/sec                  receiver
-----------------------------------------------------------

Client:

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.01 GBytes  2.58 Gbits/sec    0            sender
[  5]   0.00-10.00  sec  3.00 GBytes  2.58 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  13.7 GBytes  11.8 Gbits/sec    0            sender
[  7]   0.00-10.00  sec  13.7 GBytes  11.8 Gbits/sec                  receiver
[  9]   0.00-10.00  sec  2.74 GBytes  2.35 Gbits/sec    0            sender
[  9]   0.00-10.00  sec  2.73 GBytes  2.35 Gbits/sec                  receiver
[ 11]   0.00-10.00  sec  2.98 GBytes  2.56 Gbits/sec    0            sender
[ 11]   0.00-10.00  sec  2.98 GBytes  2.56 Gbits/sec                  receiver
[ 13]   0.00-10.00  sec  13.7 GBytes  11.8 Gbits/sec    0            sender
[ 13]   0.00-10.00  sec  13.7 GBytes  11.8 Gbits/sec                  receiver
[ 15]   0.00-10.00  sec  2.79 GBytes  2.39 Gbits/sec    0            sender
[ 15]   0.00-10.00  sec  2.78 GBytes  2.39 Gbits/sec                  receiver
[ 17]   0.00-10.00  sec  2.88 GBytes  2.47 Gbits/sec    0            sender
[ 17]   0.00-10.00  sec  2.88 GBytes  2.47 Gbits/sec                  receiver
[ 19]   0.00-10.00  sec  13.7 GBytes  11.8 Gbits/sec    0            sender
[ 19]   0.00-10.00  sec  13.7 GBytes  11.8 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  55.5 GBytes  47.7 Gbits/sec    0             sender
[SUM]   0.00-10.00  sec  55.5 GBytes  47.6 Gbits/sec                  receiver

• As soon as RKE2 is setup on Control plane node -

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  2.32 GBytes  1.99 Gbits/sec                  receiver
[  8]   0.00-10.00  sec  6.48 GBytes  5.56 Gbits/sec                  receiver
[ 10]   0.00-10.00  sec  2.42 GBytes  2.08 Gbits/sec                  receiver
[ 12]   0.00-10.00  sec  2.79 GBytes  2.40 Gbits/sec                  receiver
[ 14]   0.00-10.00  sec  1.79 GBytes  1.54 Gbits/sec                  receiver
[ 16]   0.00-10.00  sec  4.40 GBytes  3.78 Gbits/sec                  receiver
[ 18]   0.00-10.00  sec  4.25 GBytes  3.65 Gbits/sec                  receiver
[ 20]   0.00-10.00  sec  2.20 GBytes  1.89 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  26.7 GBytes  22.9 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201 (test #2)
-----------------------------------------------------------

Client :

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.32 GBytes  1.99 Gbits/sec  2036            sender
[  5]   0.00-10.00  sec  2.32 GBytes  1.99 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  6.48 GBytes  5.57 Gbits/sec  1178            sender
[  7]   0.00-10.00  sec  6.48 GBytes  5.56 Gbits/sec                  receiver
[  9]   0.00-10.00  sec  2.42 GBytes  2.08 Gbits/sec  1447            sender
[  9]   0.00-10.00  sec  2.42 GBytes  2.08 Gbits/sec                  receiver
[ 11]   0.00-10.00  sec  2.80 GBytes  2.40 Gbits/sec  570            sender
[ 11]   0.00-10.00  sec  2.79 GBytes  2.40 Gbits/sec                  receiver
[ 13]   0.00-10.00  sec  1.79 GBytes  1.54 Gbits/sec  545            sender
[ 13]   0.00-10.00  sec  1.79 GBytes  1.54 Gbits/sec                  receiver
[ 15]   0.00-10.00  sec  4.41 GBytes  3.79 Gbits/sec  517            sender
[ 15]   0.00-10.00  sec  4.40 GBytes  3.78 Gbits/sec                  receiver
[ 17]   0.00-10.00  sec  4.25 GBytes  3.65 Gbits/sec  1109            sender
[ 17]   0.00-10.00  sec  4.25 GBytes  3.65 Gbits/sec                  receiver
[ 19]   0.00-10.00  sec  2.20 GBytes  1.89 Gbits/sec  630            sender
[ 19]   0.00-10.00  sec  2.20 GBytes  1.89 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  26.7 GBytes  22.9 Gbits/sec  8032             sender
[SUM]   0.00-10.00  sec  26.7 GBytes  22.9 Gbits/sec                  receiver

iperf Done.

Kindly let me know if any CNI plug-in or network setting needs to be tweaked, I have tried this with different combination but same result. As soon as I removed RKE2 and reboot system I started to see original performance

[ram-suradkar]

Additional setup details

No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 12 (bookworm)
Release:	12
Codename:	bookworm

# iptables -v
iptables v1.8.9

[manuelbuil]

Where is the client running? Where is the server running? What kind of CNI are you deploying with?

[manuelbuil]

You are seeing this behaviour with Canal and Calico, right? Could you check with Cilium? I am very surprised

[ram-suradkar]

exactly my reaction and not something differently configured just followed - https://docs.rke2.io/install/quickstart guide.
yes calico and canal

[ram-suradkar]

should I use -

# /var/lib/rancher/rke2/server/manifests/rke2-cilium-config.yaml
---
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-cilium
  namespace: kube-system
spec:
  valuesContent: |-
    kubeProxyReplacement: true
    k8sServiceHost: "localhost"
    k8sServicePort: "6443"

from doc for setting up cilium ?

[ram-suradkar]

I tried with cilium as well -

• Before installation :

root@k8s-control:~/iperf/src# date
Wed Feb 26 04:24:13 AM EST 2025
root@k8s-control:~/iperf/src# ./iperf3 -s
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 172.x.x.x, port 42716
[  5] local 172.x.x.x port 5201 connected to 172.x.x.x port 42720
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  6.79 GBytes  58.3 Gbits/sec
[  5]   1.00-2.00   sec  6.62 GBytes  56.9 Gbits/sec
[  5]   2.00-3.00   sec  5.83 GBytes  50.1 Gbits/sec
[  5]   3.00-4.00   sec  5.92 GBytes  50.8 Gbits/sec
[  5]   4.00-5.00   sec  6.40 GBytes  55.0 Gbits/sec
[  5]   5.00-6.00   sec  6.22 GBytes  53.4 Gbits/sec
[  5]   6.00-7.00   sec  6.45 GBytes  55.4 Gbits/sec
[  5]   7.00-8.00   sec  5.88 GBytes  50.5 Gbits/sec
[  5]   8.00-9.00   sec  5.79 GBytes  49.8 Gbits/sec
[  5]   9.00-10.00  sec  5.58 GBytes  48.0 Gbits/sec
[  5]  10.00-10.00  sec  2.50 MBytes  26.4 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  61.5 GBytes  52.8 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201 (test #2)
-----------------------------------------------------------

===== node 2========

root@k8s-worker:~/iperf/src# date
Wed Feb 26 04:24:24 AM EST 2025
root@k8s-worker:~/iperf/src# ./iperf3 -c 172.x.x.x -t 10
Connecting to host 172.x.x.x, port 5201
[  5] local 172.x.x.x port 42720 connected to 172.x.x.x port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  6.79 GBytes  58.3 Gbits/sec    0   3.12 MBytes
[  5]   1.00-2.00   sec  6.63 GBytes  56.9 Gbits/sec    0   3.12 MBytes
[  5]   2.00-3.00   sec  5.83 GBytes  50.1 Gbits/sec    0   3.12 MBytes
[  5]   3.00-4.00   sec  5.92 GBytes  50.8 Gbits/sec    0   3.12 MBytes
[  5]   4.00-5.00   sec  6.40 GBytes  55.0 Gbits/sec    0   3.12 MBytes
[  5]   5.00-6.00   sec  6.22 GBytes  53.4 Gbits/sec    0   3.12 MBytes
[  5]   6.00-7.00   sec  6.45 GBytes  55.4 Gbits/sec    0   3.12 MBytes
[  5]   7.00-8.00   sec  5.88 GBytes  50.5 Gbits/sec    0   3.12 MBytes
[  5]   8.00-9.00   sec  5.80 GBytes  49.8 Gbits/sec    0   3.12 MBytes
[  5]   9.00-10.00  sec  5.58 GBytes  47.9 Gbits/sec    0   3.12 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  61.5 GBytes  52.8 Gbits/sec    0            sender
[  5]   0.00-10.00  sec  61.5 GBytes  52.8 Gbits/sec                  receiver

iperf Done.

• Install RKE2 with cilium CNI :

root@k8s-control:~/iperf/src# date
Wed Feb 26 04:27:07 AM EST 2025
root@k8s-control:~/iperf/src# mkdir -p /etc/rancher/rke2/
root@k8s-control:~/iperf/src# vim /etc/rancher/rke2/config.yaml
root@k8s-control:~/iperf/src# mkdir -p /var/lib/rancher/rke2/server/manifests/
root@k8s-control:~/iperf/src# vim /var/lib/rancher/rke2/server/manifests/rke2-cilium-config.yaml
root@k8s-control:~/iperf/src# curl -sfL https://get.rke2.io | sudo sh -
[INFO]  finding release for channel stable
[INFO]  using v1.31.5+rke2r1 as release
[INFO]  downloading checksums at https://github.com/rancher/rke2/releases/download/v1.31.5+rke2r1/sha256sum-amd64.txt
[INFO]  downloading tarball at https://github.com/rancher/rke2/releases/download/v1.31.5+rke2r1/rke2.linux-amd64.tar.gz
[INFO]  verifying tarball
[INFO]  unpacking tarball file to /usr/local
root@k8s-control:~/iperf/src# systemctl enable rke2-server
Created symlink /etc/systemd/system/multi-user.target.wants/rke2-server.service → /usr/local/lib/systemd/system/rke2-server.service.
root@k8s-control:~/iperf/src# systemctl start rke2-server




root@k8s-control:~/iperf/src# date
Wed Feb 26 04:37:07 AM EST 2025
root@k8s-control:~/iperf/src# kubectl get nodes
NAME          STATUS   ROLES                       AGE     VERSION
k8s-control   Ready    control-plane,etcd,master   3m40s   v1.31.5+rke2r1
k8s-worker    Ready    <none>                      58s     v1.31.5+rke2r1
root@k8s-control:~/iperf/src# ./iperf3 -s
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 172.x.x.x, port 60214
[  5] local 172.x.x.x port 5201 connected to 172.x.x.x port 60230
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.96 GBytes  16.8 Gbits/sec
[  5]   1.00-2.00   sec  2.03 GBytes  17.5 Gbits/sec
[  5]   2.00-3.00   sec  2.09 GBytes  18.0 Gbits/sec
[  5]   3.00-4.00   sec  2.03 GBytes  17.5 Gbits/sec
[  5]   4.00-5.00   sec  2.06 GBytes  17.7 Gbits/sec
[  5]   5.00-6.00   sec  1.89 GBytes  16.2 Gbits/sec
[  5]   6.00-7.00   sec  2.07 GBytes  17.8 Gbits/sec
[  5]   7.00-8.00   sec  2.13 GBytes  18.3 Gbits/sec
[  5]   8.00-9.00   sec  2.31 GBytes  19.8 Gbits/sec
[  5]   9.00-10.00  sec  2.38 GBytes  20.4 Gbits/sec
[  5]  10.00-10.00  sec  1.00 MBytes  21.1 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  21.0 GBytes  18.0 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201 (test #2)
-----------------------------------------------------------
^Ciperf3: interrupt - the server has terminated by signal Interrupt(2)

[manuelbuil]

@thomasferrandiz any idea?