Netbird seems to "die" on my server

[GronbergInfo]

I run
sudo netbird down && sudo netbird up --allow-server-ssh

On the server, and it starts to work, then after a uncertain time, it just stops communicate, resolve dns and such until i run that command again

client and server has netbird 0.35.0.

If i ssh into the server using lan ip and check netbird status, it looks ok/normal

> netbird status
OS: linux/amd64
Daemon version: 0.35.0
CLI version: 0.35.0
Management: Connected
Signal: Connected
Relays: 3/3 Available
Nameservers: 1/1 Available
FQDN: <servername>.netbird.cloud
NetBird IP: 100.85.x.y/16
Interface type: Kernel
Quantum resistance: true
Routes: -
Networks: -
Peers count: 2/2 Connected

EDIT: just added 2 peers (laptop + android tablet) and it died for the server again.

[mlsmaycon]

Hello @GronbergInfo thanks for reporting the issue, can you please run the following commands on the server and one of the clients?

Enable debug information

# enable trace logs:
netbird debug log level trace
# enable network map report
netbird debug persistence on

When the issue happens again:

netbird debug bundle -SA

This will generate an anonymized bundle file with log information. Then, please share them with us.

[mlsmaycon]

Also, can you confirm that the issue happens on SSH connections with the server or, in general, the peer connection 'dies' and you can't even ping it? and how long does it take to have issues?

[GronbergInfo]

Hello @GronbergInfo thanks for reporting the issue, can you please run the following commands on the server and one of the clients?

Enable debug information

# enable trace logs:
netbird debug log level trace
# enable network map report

netbird debug persistence on

When the issue happens again:

netbird debug bundle -SA

This will generate an anonymized bundle file with log information. Then, please share them with us.

From client: netbird.debug.3148610165.zip
netbird.debug.3148610165.zip

From Server: netbird.debug.1025569393.zip
netbird.debug.1025569393.zip

[GronbergInfo]

Also, can you confirm that the issue happens on SSH connections with the server or, in general, the peer connection 'dies' and you can't even ping it? and how long does it take to have issues?

It seems everything "dies" but hostname lookup

> nslookup hades10gbe
Server:         100.85.228.40
Address:        100.85.228.40#53

Non-authoritative answer:
Name:   hades10gbe.netbird.cloud
Address: 100.85.119.90
** server can't find hades10gbe.netbird.cloud: NXDOMAIN

[mlsmaycon]

@GronbergInfo it doesn't seem like you executed the commands as shared in #3107 (comment). Can you please do the exact steps?

Also, can you provide an answer to the following questions?

1. when the connection dies, you can run netbird status -d, and you will see a connection to signal and management, but no connection to the peer?
2. how long until the issue happens?
3. Does the issue happen if you have rosenpass integration disabled?

[GronbergInfo]

@GronbergInfo it doesn't seem like you executed the commands as shared in #3107 (comment). Can you please do the exact steps?

Also, can you provide an answer to the following questions?

1. when the connection dies, you can run netbird status -d, and you will see a connection to signal and management, but no connection to the peer?
2. how long until the issue happens?
3. Does the issue happen if you have rosenpass integration disabled?

nbd.txt - client
nbd2.txt - server
nbd.txt
nbd2.txt

2. it feels random, can work for 15-30-60 minutes, uncertain, it can stop working within a heart beat almost.

3. Disabled rosenpass on client and server, mounted an nfs share, started to copy a 600GB vm vmware folder, changed group for my android tablet and applied ACL for this group. After 10 or so minutes, it is still running ok.

[GronbergInfo]

still 15 or so minutes later, it still up and running.
So it seems to be attached to the rosenpass function.

[mlsmaycon]

It is caused by the key not being rotated due to a firewall issue. We are missing a default allow rule for rosenpass.

If this continues, can you add a UDP rule like the one in the image and try again? We will be working on a permanent fix.

[GronbergInfo]

It is caused by the key not being rotated due to a firewall issue. We are missing a default allow rule for rosenpass.

If this continues, can you add a UDP rule like the one in the image and try again? We will be working on a permanent fix.

Even with this setting, it seems it dies with quantum enabled.

[mlsmaycon]

Thanks @GronbergInfo , can you please enable the logs and share them with us once the connection fail? we are running the same test on our side too.