Netbird constantly vomits STUN packets causing absurdly high network activity

[coredoesdev]

Describe the problem

Any running NetBird client is spewing out hundreds of STUN packets per second, crippling several networks we have dev machines at.

To Reproduce

Steps to reproduce the behavior:

1. Run a NetBird client
2. Observe the higher-than-baseline network usage
3. (Optional) Open WireShark, or similar network capture tool, and notice the absurd amount of STUN packets being sent out.

Expected behavior

Once a tunnel has been successfully established or if a tunnel has failed to connect for a long enough period of time, calm down the STUN packet flow, perhaps to 1 packet per 2 seconds per host.

Are you using NetBird Cloud?

Selfhosted

NetBird version

0.26.2

NetBird status -d output:

Peers detail:
 east2.xh:
  NetBird IP: 100.78.27.28
  Public key: Ak1eYFgHtzF08ZNHxdCmwbMxviE6v3YkkXKKIOiVaTQ=
  Status: Connected
  -- detail --
  Connection type: Relayed
  Direct: false
  ICE candidate (Local/Remote): relay/host
  ICE candidate endpoints (Local/Remote): REMOVED
  Last connection update: 2024-03-21 11:57:18
  Last WireGuard handshake: 2024-03-21 12:00:21
  Transfer status (received/sent) 588 B/632 B
  Quantum resistance: false

 terra-twr.xh:
  NetBird IP: 100.78.61.217
  Public key: bXeYvvifYllBoYFCEFvf7T7PDfmHS21DlYr5rwpObWU=
  Status: Connected
  -- detail --
  Connection type: Relayed
  Direct: false
  ICE candidate (Local/Remote): relay/srflx
  ICE candidate endpoints (Local/Remote): REMOVED
  Last connection update: 2024-03-21 11:57:18
  Last WireGuard handshake: 2024-03-21 12:00:16
  Transfer status (received/sent) 440 B/456 B
  Quantum resistance: false

 terra-pi4.xh:
  NetBird IP: 100.78.94.160
  Public key: 6s+peKaEKNQazQOEGqeeei/w3u/vA6FuUZUyGwV1blI=
  Status: Connected
  -- detail --
  Connection type: Relayed
  Direct: false
  ICE candidate (Local/Remote): relay/srflx
  ICE candidate endpoints (Local/Remote): REMOVED
  Last connection update: 2024-03-21 11:57:18
  Last WireGuard handshake: 2024-03-21 11:59:28
  Transfer status (received/sent) 376 B/584 B
  Quantum resistance: false

 terra-ip12m.xh:
  NetBird IP: 100.78.100.182
  Public key: d8xHD1jw90V6FoqR2u3dnVQuqxR+h4EnY0vaaCQsYjM=
  Status: Disconnected
  -- detail --
  Connection type: Relayed
  Direct: false
  ICE candidate (Local/Remote): relay/host
  ICE candidate endpoints (Local/Remote): REMOVED
  Last connection update: -
  Last WireGuard handshake: 2024-03-21 11:59:28
  Transfer status (received/sent) 376 B/584 B
  Quantum resistance: false

 east1.xh:
  NetBird IP: 100.78.104.214
  Public key: ec7QLhyyGfk5N4LTUvj+JJ4g0KJangpSP1axbrArNkY=
  Status: Connected
  -- detail --
  Connection type: Relayed
  Direct: false
  ICE candidate (Local/Remote): relay/host
  ICE candidate endpoints (Local/Remote): REMOVED
  Last connection update: 2024-03-21 11:57:18
  Last WireGuard handshake: 2024-03-21 11:59:28
  Transfer status (received/sent) 216 B/616 B
  Quantum resistance: false

 terra-fw13.xh:
  NetBird IP: 100.78.105.61
  Public key: uFLayqa7vL8Ad9nxqeDItuHhK0RrcpX6Cujl82I1gBs=
  Status: Disconnected
  -- detail --
  Connection type: Relayed
  Direct: false
  ICE candidate (Local/Remote): relay/host
  ICE candidate endpoints (Local/Remote): REMOVED
  Last connection update: 2024-03-21 08:58:41
  Last WireGuard handshake: 2024-03-21 11:59:28
  Transfer status (received/sent) 376 B/584 B
  Quantum resistance: false

 de-fsn1.xh:
  NetBird IP: 100.78.113.56
  Public key: 92eyOPcPChnMSY3DGj4Ck3v20iRwTfXDnXBvAjVHKAE=
  Status: Connected
  -- detail --
  Connection type: Relayed
  Direct: false
  ICE candidate (Local/Remote): relay/host
  ICE candidate endpoints (Local/Remote): REMOVED
  Last connection update: 2024-03-21 11:57:19
  Last WireGuard handshake: 2024-03-21 11:59:28
  Transfer status (received/sent) 376 B/584 B
  Quantum resistance: false

 us-sfo1.xh:
  NetBird IP: 100.78.137.206
  Public key: D6q2kLm8YJpBS5q6cJly2Skrq6BoJvdoXGWkU1aLOyw=
  Status: Connected
  -- detail --
  Connection type: Relayed
  Direct: false
  ICE candidate (Local/Remote): relay/host
  ICE candidate endpoints (Local/Remote): REMOVED
  Last connection update: 2024-03-21 11:57:18
  Last WireGuard handshake: 2024-03-21 11:59:27
  Transfer status (received/sent) 248 B/584 B
  Quantum resistance: false

 fxtwr.xh:
  NetBird IP: 100.78.145.187
  Public key: nILKxAfT1lc3/tG2jJFeBckyydBJ/9Tv5ABU1m1Xik4=
  Status: Connected
  -- detail --
  Connection type: Relayed
  Direct: false
  ICE candidate (Local/Remote): relay/srflx
  ICE candidate endpoints (Local/Remote): REMOVED
  Last connection update: 2024-03-21 11:57:18
  Last WireGuard handshake: 2024-03-21 11:59:26
  Transfer status (received/sent) 616 B/556 B
  Quantum resistance: false

 core-ipxr.xh:
  NetBird IP: 100.78.155.157
  Public key: QSOeszg4L/gC/HcXbjGveORxTeVNditQMcjCh1IduWM=
  Status: Disconnected
  -- detail --
  Connection type: Relayed
  Direct: false
  ICE candidate (Local/Remote): relay/srflx
  ICE candidate endpoints (Local/Remote): REMOVED
  Last connection update: -
  Last WireGuard handshake: 2024-03-21 12:00:18
  Transfer status (received/sent) 472 B/360 B
  Quantum resistance: false

 central1.xh:
  NetBird IP: 100.78.185.38
  Public key: SZAef/LuGbjGoHQSusklQDK2FafOzQXkXEXAtTrzKFc=
  Status: Connected
  -- detail --
  Connection type: Relayed
  Direct: false
  ICE candidate (Local/Remote): relay/srflx
  ICE candidate endpoints (Local/Remote): REMOVED
  Last connection update: 2024-03-21 11:57:18
  Last WireGuard handshake: 2024-03-21 12:00:15
  Transfer status (received/sent) 472 B/360 B
  Quantum resistance: false

 core-twr.xh:
  NetBird IP: 100.78.199.227
  Public key: JV/hVFs+PiQpn/4i25qykYsb1I1HZo0OlYT5C2f/oV0=
  Status: Connected
  -- detail --
  Connection type: Relayed
  Direct: false
  ICE candidate (Local/Remote): relay/srflx
  ICE candidate endpoints (Local/Remote): REMOVED
  Last connection update: 2024-03-21 11:57:18
  Last WireGuard handshake: 2024-03-21 12:00:18
  Transfer status (received/sent) 472 B/360 B
  Quantum resistance: false

 in-blr1.xh:
  NetBird IP: 100.78.202.128
  Public key: 2y4FpWgfpv1YshViYvOlHGXZYJgAo3M2ny33Gap6EVo=
  Status: Connected
  -- detail --
  Connection type: Relayed
  Direct: false
  ICE candidate (Local/Remote): relay/host
  ICE candidate endpoints (Local/Remote): REMOVED
  Last connection update: 2024-03-21 11:57:19
  Last WireGuard handshake: 2024-03-21 11:59:28
  Transfer status (received/sent) 312 B/520 B
  Quantum resistance: false

 sg-sgp1.xh:
  NetBird IP: 100.78.220.249
  Public key: cmKbR1Z6OcLvpM1phpd8msAioq7NnnJSSSCWzCQ2Ok0=
  Status: Connected
  -- detail --
  Connection type: Relayed
  Direct: false
  ICE candidate (Local/Remote): relay/host
  ICE candidate endpoints (Local/Remote): REMOVED
  Last connection update: 2024-03-21 11:57:19
  Last WireGuard handshake: 2024-03-21 11:59:28
  Transfer status (received/sent) 312 B/520 B
  Quantum resistance: false

 east3.xh:
  NetBird IP: 100.78.250.217
  Public key: PhXzdh9pTN/Ika3FYfrbSjbVli+Yaw3ejcnNHAi4jBA=
  Status: Connected
  -- detail --
  Connection type: Relayed
  Direct: false
  ICE candidate (Local/Remote): relay/host
  ICE candidate endpoints (Local/Remote): REMOVED
  Last connection update: 2024-03-21 11:57:18
  Last WireGuard handshake: 2024-03-21 11:59:57
  Transfer status (received/sent) 16.4 KiB/12.1 KiB
  Quantum resistance: false

Daemon version: 0.26.2
CLI version: 0.26.2
Management: Connected to https://REMOVED:443
Signal: Connected to https://REMOVED:443
Relays: 
  [stun:REMOVED:3478] is Available
  [turn:REMOVED:3478?transport=udp] is Available
FQDN: core-e14.xh
NetBird IP: 100.78.3.100/16
Interface type: Kernel
Quantum resistance: false
Peers count: 12/15 Connected

Screenshots

n/a

Additional context

Occurs on all devices on all networks.

[lixmal]

The STUN packets you are seeing are the relayed VPN traffic. This is expected because your peer seems to connect to all other peers via the relay. You should investigate if that machines sits behind a very restrictive NAT that requires to use the relay instead of peer to peer connections

[coredoesdev]

This machine is behind a NAT, but it shouldn’t prevent establishing a direct connection… standard traversal techniques should be succeeding - relaying shouldn’t be nessecary. I’ll look on other machines and see what they’re emitting… the common factor is that a running NetBird client has a habit of crippling networks. I’ll investigate further.

[nazarewk]

@coredoesdev are you still having issues with latest NetBird versions? There were a lot of connectivity establishment improvements since opening the issue

[coredoesdev]

We ended up switching away from Netbird. Feel free to close this out