Broken DNS resolution of *.our.domain on Windows client

[tomashora]

Describe the problem

The resolution of all subdomains under *our.domain does not work for certain application (for example any web browser or ping tool, however nslookup resolves IP correctly). This used to happen in the past when netbird was shut down incorrectly, as discussed on Slack. Now it seems to happen the same way - hard laptop shutdown, system boots up, dns not resolved.

This results that the clients cannot connect.

To Reproduce

Steps to reproduce the behavior:
TBD

Expected behavior

All DNS records should be resolved correctly.

Are you using NetBird Cloud?

Self-hosted (v0.31.1 incl. relay as well as coturn)

NetBird version

0.31.1

NetBird status -dA output:

X

Do you face any (non-mobile) client issues?

2024-11-15T08:27:16+01:00 ERRO util/grpc/dialer.go:38: Failed to dial: dial: dial tcp: lookup netbird.our.domain: no such host

Screenshots

X

Additional context

The easiest to fix it is to connecto to the Netbird Cloud instance, which somehows resets the windows DNS configutation so the *.our.domain is immediately resolved correctly.

Output of Resolve-DnsName -Name www.our.domain
Resolve-DnsName: www.unipi.technology : Daná operace se vrátila, protože vypršel časový limit. //Time exceeded

Output of: ping www.our.domain
Ping request could not find host www.our.domain. Please check the name and try again.

Output of: nslookup www.our.domain

Server:  dns.google
Address:  8.8.8.8

Non-authoritative answer:
Name:    our.domain
Address:  correct IP address
Aliases:  www.our.domain

Output of Get-DnsClientNrptPolicy

Namespace                        : .ourdomain.local
QueryPolicy                      :
SecureNameQueryFallback          :
DirectAccessIPsecCARestriction   :
DirectAccessProxyName            :
DirectAccessDnsServers           :
DirectAccessEnabled              :
DirectAccessProxyType            : NoProxy
DirectAccessQueryIPsecEncryption :
DirectAccessQueryIPsecRequired   : False
NameServers                      : 10.220.255.254
DnsSecIPsecCARestriction         :
DnsSecQueryIPsecEncryption       :
DnsSecQueryIPsecRequired         : False
DnsSecValidationRequired         : False
NameEncoding                     : Utf8WithoutMapping

Namespace                        : .our.domain
QueryPolicy                      :
SecureNameQueryFallback          :
DirectAccessIPsecCARestriction   :
DirectAccessProxyName            :
DirectAccessDnsServers           :
DirectAccessEnabled              :
DirectAccessProxyType            : NoProxy
DirectAccessQueryIPsecEncryption :
DirectAccessQueryIPsecRequired   : False
NameServers                      : 10.220.255.254
DnsSecIPsecCARestriction         :
DnsSecQueryIPsecEncryption       :
DnsSecQueryIPsecRequired         : False
DnsSecValidationRequired         : False
NameEncoding                     : Utf8WithoutMapping

[roberthase]

try deleting this registry-key, when this happens. Computer\HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\Dnscache\Parameters\DnsPolicyConfig\NetBird-Match

can you also confirm that the domain used for your netbird controller is also supposed to be routed to through the wireguard tunnel after the connection is established?

[cleveHEX]

@roberthase thanks, this did the trick, is it possible to be done by Netbird service, as this requires elevated permissions?
It behaved rather weirdly, before I did your trick, but I am not an expert on how networking works in Windows. nslookup returned a proper result from the main DNS resolver, but ping or traceroute failed.

Not sure if I get your question, but the management and admin domains are a subdomains (management.example.com) of the domain that was stuck in the registry (.example.com)

[roberthase]

The registry key i posted gets created when netbird successfully connects.

All matched domains you configured in your controller under DNS -> Nameservers are listed here.

With matched domains configured, every domain you entered can only be accessible over the wireguard tunnel/interface.

When netbird is gracefully shutdown/disconnected, the registry gets deleted.

There can be instances, where your windows os could not shutdown correctly and thats where things get ugly, if the domain of your controller is also a matched domain.

You boot your system and the registry key is still there and now your netbird-client can't reach your controller netbird.example.com, because example.com is supposed to go through your wireguard tunnel/interface.

Nslookup should give you the right result, because its using the dns server configured on your pc or your router, but the routing is wrong.

[cleveHEX]

Thank you for the explanation. I have noticed that Netbird removes those entries on graceful deactivation. I thought if Netbird could try to delete this entry also on its start (pre-start clean up, before it starts actually doing something).

Anyways, I will propose to move the management out of the domains that go via wireguard. Unfortunately this will mean a lot of changes (management URL will stay the same, but domains and services will be buried one level lower under one more subdomain).

[tomashora]

It's still happening randomly also with client 0.35.2

[roberthase]

we route only relevant subdomains now, so netbird.example.com is not affected. maybe this is also possible for your enviroment.

[jakovnikolic]

It's still happening randomly also with client 0.35.2

We are experiencing the same issue with latest client version. Could this be corrected so that netbird client takes care of cleanup each time host is started or as @cleveHEX has suggested on client start?

[lixmal]

@jakovnikolic both of these are implemented. Can you share more about the issue you have?

[jakovnikolic]

This morning i have got support request from 2 of my colleagues telling me they are not able to connect or access any of our internal domains. They have been using Windows operating system and they have just updated clients to latest version v0.35.2.

After manual removal of HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\Dnscache\Parameters\DnsPolicyConfig\NetBird-Match they have managed to connect and access everything as expected.

We have tested running nslookup and domain of our VPN server resolves without any issues but using ping we would get no response. Since nslookup opens a winsock connection on the DNS port and issues a query, whereas ping uses the DNS Client service.

[tomashora]

This morning i have got support request from 2 of my colleagues telling me they are not able to connect or access any of our internal domains. They have been using Windows operating system and they have just updated clients to latest version v0.35.2.

After manual removal of HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\Dnscache\Parameters\DnsPolicyConfig\NetBird-Match they have managed to connect and access everything as expected.

We have tested running nslookup and domain of our VPN server resolves without any issues but using ping we would get no response. Since nslookup opens a winsock connection on the DNS port and issues a query, whereas ping uses the DNS Client service.

Exactly the same for us

[lixmal]

Can you share the contents of %PROGRAMDATA\Netbird\state.json the next time this happens before starting netbird?

Does this happen on boot or when waking up from sleep?
If after sleep, does rebooting fix the issue (netbird cleans up on service start)?

Does netbird state clean dns_state fix it?
Does netbird service restart fix it?

Also logs would be helpful, e.g. with netbird debug bundle -A once netbird is running again

[Amit-Tomar-Livspace]

@roberthase I am seeing same issue on macOS, what is the alternate of registry clearing on MacOS ?

[roberthase]

sorry. i do not know where these settings are stored on MacOS.

To follow up with a new issues we experienced after upgrading clients, routingpeers and the controller from 0.31.0 0.35.2.

On some Windows 11 Clients the registry key for the routes is created after the connection is established and then is immediately deleted afterwards.

So these clients can connect to the controller but not the routes.

Downgrading or uninstalling/reinstalling the client to 0.31.0 has no effect. The new issues persists.

Any advice on how to remedy this issue?

[Amit-Tomar-Livspace]

For anyone's future reference, this is how I resolved temporarily on MacOS:

1. Open the file sudo vim /etc/hosts.
2. Manually find the IP address of your company domain which is not getting resolved.
3. At the end of above file, add an entry corresponding to your data so that DNS resolution can happen.
   eg.

99.22.11.33 foo.mydomain.com

Once this was done, netbird started working properly. After this I removed the entry from hosts file and netbird continues to work fine. Adding this entry anywhere else in dns resolution setting was not working. I believe hosts file takes preference over everything else and hence it worked.