pfSense seems to be blocking connections

[johncarterofmars]

Describe the problem
I have a private LAN, 192.168.200.0/24. I have 2 devices on this, both Linux. One is just a peer, while the other, an Ubuntu 20.04 VM, is a routing host. These are both behind a pfSense firewall. Both of these can connect to the controller at a self-hosted VM on AWS. Obviously, these two devices can connect to each other as they are on the same LAN. If I do netbird status (full details below), it shows that the desktop is connected to the routing node (keeper).
However, on the laptop at another location, any location that is behind a pfsense firewall, the laptop will connect to the AWS hosted controller, but it is unable to connect to any other peer.
Laptop cannot ping either of the other two (as it is not connected) and these 2 cannot ping the laptop. I get this message:
Destination Host Unreachable ping: sendmsg: Required key not available
I am sure this has something to do with how pfsense handles the returning wireguard traffic. I am not asking to help troubleshoot the pfsense part, its more about what direction to even go to do it. Is there a way to set static ports so I can configure the NAT in the firewall to those static ports? Is anyone else having issues with NAT and connections between other sites that are behind firewalls?

To Reproduce
Add a peer behind a pfsense firewall
Add another peer behind a pfsense firewall on another network
Ping to each other

Expected behavior
to be able to have devices behind different firewalls able to communicate with each other. I am looking for suggestions to help me figure out what the issue may be so I can resolve it.

NetBird status -d output:

sudo netbird status -d
Peers detail:
laptop.netbird.selfhosted:
NetBird IP: 100.114.155.201
Public key: ppCYTEvMqrAIo6sCrj9euH7t1zTnERL4uCFX2Bh6sU0=
Status: Connecting
-- detail --
Connection type: P2P
Direct: false
ICE candidate (Local/Remote): srflx/host
Last connection update: 2024-01-01 09:41:25

keeper.netbird.selfhosted:
NetBird IP: 100.114.255.6
Public key: rit7uH0pG4kQb0owERzvoLwq84gzAk/dr354fSLV+QU=
Status: Connected
-- detail --
Connection type: P2P
Direct: true
ICE candidate (Local/Remote): srflx/host
Last connection update: 2024-01-01 09:29:08

Daemon version: 0.25.2
CLI version: 0.25.2
Management: Connected to https://REDACTED_URL
Signal: Connected to https://REDACTED_UR
FQDN: desktop.netbird.selfhosted
NetBird IP: 100.112.196.218/16
Interface type: Kernel
Peers count: 1/2 Connected

Screenshots

Additional context

[johncarterofmars]

Does anyone have any suggestions on what I need to ensure is opened up on either incoming or outgoing traffic? It seems the NATing is causing the issue.

[Vignesh1230]

I've also had the same issue as im installing Netbird for the first time.
My setup:
EC2 Instance running Netbird self hosted.
Local network pfSense firewall.
A CentOS VM as a routing host on local network.
Try connect using Cellular on iphone, but 0 of 2 peers connected.

user@tailscale-vpn:~$ sudo netbird status -d
Peers detail:
iphone.netbird.selfhosted:
NetBird IP: 100.115.74.205
Public key: yPh2oHrVVYesmY4R0KxCLz3at2kmyMJdCDNlHALh02A=
Status: Disconnected
-- detail --
Connection type:
Direct: false
ICE candidate (Local/Remote): -/-
Last connection update: 2024-01-08 14:37:01

mbp.netbird.selfhosted:
NetBird IP: 100.115.117.141
Public key: 4VvAVFSblYeSFMKDblMkK2Yook7hQAV5gGM+TLsz4VQ=
Status: Disconnected
-- detail --
Connection type:
Direct: false
ICE candidate (Local/Remote): -/-
Last connection update: 2024-01-08 14:28:46

Daemon version: 0.25.3
CLI version: 0.25.3
Management: Connected to https://REDACTED_URL
Signal: Connected to https://REDACTED_URL
FQDN: tailscale-vm.netbird.selfhosted
NetBird IP: 100.115.51.88/16
Interface type: Kernel
Peers count: 0/2 Connected

user@tailscale-vm~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 82:c5:44:47:ff:6c brd ff:ff:ff:ff:ff:ff
inet 10.0.1.82/20 brd 10.0.15.255 scope global dynamic ens18
valid_lft 4888sec preferred_lft 4888sec
inet6 fe80::80c5:44ff:fe47:ff6c/64 scope link
valid_lft forever preferred_lft forever
3: tailscale0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc fq_codel state UNKNOWN group default qlen 500
link/none
inet 100.69.15.30/32 scope global tailscale0
valid_lft forever preferred_lft forever
inet6 fd7a:115c:a1e0::2b85:f1e/128 scope global
valid_lft forever preferred_lft forever
inet6 fe80::91fd:edc4:c7d5:ea25/64 scope link stable-privacy
valid_lft forever preferred_lft forever
6: netmaker: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
link/none
inet 10.20.30.2/24 brd 10.20.30.255 scope global netmaker
valid_lft forever preferred_lft forever
9: wt0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1280 qdisc noqueue state UNKNOWN group default qlen 1000
link/none
inet 100.115.51.88/16 brd 100.115.255.255 scope global wt0
valid_lft forever preferred_lft forever

user@tailscale-vm:~$ ping 100.115.51.88
PING 100.115.51.88 (100.115.51.88) 56(84) bytes of data.
^C
--- 100.115.51.88 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3053ms

Im running Tailscale flawlessly right now (with the tailscale VM as the subnet gateway).

Nothing is working..

[johncarterofmars]

I am surprised there hasn't been any response. There is some traffic that must be being filtered to prevent this from working. I found this, but it didn't help me to solve the issue yet - https://forum.netgate.com/topic/175395/routing-from-subnet-does-not-belong-to-pfsense/19

[stevo11811]

Hey there, Please try outbound Static NAT, this did the trick for me BUT this can cause issues if you are using an exit node since the ports will not be randomized outbound. This requires careful planning.

Obviously replace the subnet with whatever you are using.

[vasquezmi]

Have you been able to solve for this? Also, what are you using as the proxy manager? Been reading that gRPC needs to be configured at the reverse-proxy to route traffic between the services and client.

[johncarterofmars]

No, I added the rule in NAT mentioned above but it didnt make a change.

[SamB-GB]

Same here I tried adding the NAT outbound but that didn't help.

I have a similar rule for Tailscale and it works:

https://tailscale.com/kb/1146/pfsense

[nazarewk]

can you confirm whether this is still a valid issue on the latest versions of NetBird and pfsense?

[stevo11811]

Hello, from my end I have not had any issues in the past 6 months with Netbird, any special outbound NAT rules have been removed, the only rule that i do have is an incoming port forward to the routing peer.