FreeBSD: route manager installs /32 host routes for peer LAN addresses, shadowing mesh-routed subnets and silently blackholing them

[ivomarino]

Environment

NetBird	reproduced on 0.74.4 and 0.76.0
OS	FreeBSD 15.0-RELEASE
interface	wt0
not reproduced on	macOS and Linux clients of the same account, with the same peers and the same advertised routes

Summary

On FreeBSD, the route manager installs one /32 host route per peer LAN address,
pointed at the local host's own physical default gateway. Where that peer really is on the
local LAN this is correct. Where the peer is at another site — reached over the mesh — the
/32 is more specific than the mesh /24, so it wins, and that single address becomes
unreachable while the rest of the subnet keeps working.

The trigger condition, precisely: a peer's LAN address that falls inside a subnet the
same daemon is routing over the mesh. This is not "NetBird installs /32s" in general —
it legitimately installs them for peer endpoint (public) addresses, and those are
correct and must not be changed.

Expected

An address inside a mesh-routed subnet is reached over the mesh interface.

route get 10.2.0.10   ->   interface: wt0

Actual

route get 10.2.0.10   ->   gateway: 10.1.0.1   interface: <physical>

$ netstat -rn -f inet
Destination        Gateway            Flags     Netif
10.2.0.0/24        link#11            U         wt0        <- correct mesh route
10.2.0.10          10.1.0.1           UGH1      <physical>  <- shadows it, wins

Flags are <UP,GATEWAY,HOST,DONE,PROTO1>. RTF_PROTO1 means a userland daemon owns
it. There is no DYNAMIC flag, so these are not ICMP redirects — that hypothesis was
tested and disproved (redirects were enabled at the time, drop_redirect: 0).

Proof that NetBird owns these routes

Deleting them by hand and then stopping the daemon makes it try to flush exactly the routes
that were deleted, and fail on every one:

Error cleaning up routing: flush route manager: 7 errors occurred:
  * remove for key 10.2.0.10/32 ...
  * remove for key 10.2.0.11/32 ...
  (+5 more, all peer LAN addresses)

Cause and effect confirmed in both directions:

• restart the daemon → the /32s return and the affected address stops answering
• delete the /32s → the address answers again immediately

Why this is worse than it sounds

The failure is completely silent. Nothing logs it, nothing warns, and the symptom depends
entirely on what happens to live at the shadowed address. In our case the blackholed
address was a DNS forwarder target, so an entire DNS zone stopped resolving with no error
anywhere — diagnosed only because a human said "I can ping it but not resolve it".

The rest of the subnet kept working throughout, which is what makes it hard to spot: a
/24 that is 99% functional does not look like a routing fault.

Minimal reproduction

1. Two sites, 10.1.0.0/24 and 10.2.0.0/24.
2. A routing peer at site B advertises 10.2.0.0/24 into the mesh.
3. host-a at site A runs FreeBSD, accepts routes, and receives 10.2.0.0/24 via wt0.
4. peer-b is a mesh peer whose own LAN address is 10.2.0.10 — i.e. inside the
   advertised subnet.
5. On host-a: netstat -rn -f inet | grep 10.2.0.10

Expected: no host route. Actual: 10.2.0.10 10.1.0.1 UGH1 <physical>.

The differential that makes this a FreeBSD bug

Same NetBird account, same peers, same advertised routes:

client	holds the mesh /24	holds shadowing /32s
macOS	yes	no
Linux	yes	no
FreeBSD	yes	yes

So the correct behaviour already exists on the other platforms; the FreeBSD route manager
is the outlier. Version was excluded as a variable by reproducing on both 0.74.4 and
0.76.0.

---

Related

• FreeBSD: masquerade not functional on userspace routing peers — source IP not rewritten #5327 — FreeBSD: masquerade not functional on userspace routing peers. Different
  bug (source-IP rewriting, not route installation), but the same conclusion that
  the FreeBSD path is under-exercised.
• Allow Route to FreeBSD client #2200 — mentions NB_DISABLE_CUSTOM_ROUTING being needed to stop the client
  "messing with the routing table" on FreeBSD/OPNsense. If that flag is the
  intended escape hatch it is worth documenting; it is a blunt one here, since the
  mesh routes themselves are wanted and only the per-peer /32s are not.

What I am not asking for

The endpoint /32s are correct and should stay. The narrow ask is that a peer's
LAN address should not get a host route via the physical gateway when that
address falls inside a subnet the same client is already routing over the mesh.

[linear-code]

NET-1512

[lixmal]

This is not a bug. The host routes (exclusion routes) to other peers' P2P addresses are installed to avoid routing one peer through another.