Searchdomain not not accesible when using "Send all traffic over VPN connection"

[phatmandrake]

When "send all traffic over VPN connection" (the default) is selected and although it is shown to be selected in the UI.
While using nslookup, the searchdomain is not appended to the request. Toggling "Send all traffic over VPN connection", and restarting the VPN connection resolves this behavior.

This is happening on Catalina, Big Sur, Monterey.

[halo]

Hi @phatmandrake thank you for your report. This sounds familiar from this issue.

If you run this command in a Terminal to look at the network preferences file:

open /Library/Preferences/SystemConfiguration/preferences.plist

And you dig your way through to the actual "OverridePrimary" setting:

Then you should see that connections created with LinkLiar use an Integer 1, as per the code here. It used to be a String "1" and then the problem occurred that you're describing.

If you check the box manually and save the connection, does the OverridePrimary value change in any way?

[phatmandrake]

It's an integer 1, and the file does not change sadly. So far it is 100% reproducible that every time I delete and recreate the connection with macosvpn (which is amazing and you deserve the github nobel prize), the searchdomain fails to append until I toggle it.

I will try creating the connection manually to see if the behavior is the same.

I'm wondering if this has something do with the fact that I am adding the searchdomain by networksetup after macosvpn creates the interface, and something needs to be done to get it to set correctly when done this way.

scutil --dns confirms that the searchdomain is associated with the resolver though.

If you had a way of setting the searchdomain directly with macosvpn I'd be curious to see if it had the same problem 👀

I'll do some test and update here.

[phatmandrake]

If you make the VPN through the GUI it doesn't let you select "Send all traffic over VPN connection", you can only select it after, but so far manually creating it through the GUI hasn't had the same issue. Making the VPN inactive/active doesn't affect it.

I don't know how to read apple's docs, but theoretically it's possible to toggle it right?

https://developer.apple.com/documentation/networkextension/nevpnprotocol/3131931-includeallnetworks

Edit:

The plot thickens. Do to a related issue today with accidentally pushing a misconfigured DNSSettings configuration payload that bricked network access to a machine. (Thank god we didn't deploy it to the whole fleet 😭. Heart stopping moments.)

We couldn't find where the DNS settings were being define. scutil --dns and /etc/resolver.config, were revealing nothing, but it turns out it was being defined here:

/Library/Preferences/com.apple.networkextension.plist

I happened to notice the names of some classes in the plist that related nevpnprotocol. So I decided to recreated the VPN using MacOSVPN and the plist was updated!

And behold:

IncludeAllNetworks is false

OverridePrimary is false

However...Toggling it DOES change OverridePrimary to 1!!!

Does this help?

[halo]

You did an excellent research on the cause. Unfortunately I didn't get to look into it yet.

macosvpn uses the stone-age old SystemConfiguration framework to create network services. That's the framework that the network settings GUI is mostly built upon. It is old, not well documented and difficult to handle (as in unpredictable at times depending in which order you do things).

But Apple realized the developers needed to use VPN in their apps, so Apple created the new kid in town called Network Extention in 2019 and I was under the strong impression that it only handles IKEv2 and that for every app you publish in the app store, one VPN service may be created on the user's device. But also, all of that is irrelevant, because you didn't have to program anything but you could just us a Configuration Profile file, double-click it and then you have your IKEv2 network (which is the reason macosvpn doesn't need to handle IKEv2).

Now I saw that includeAllNetworks is part of the NetworkExtension framework and it relates to both IKEv2 and IPsec. There is much code on Github relating to includeAllNetworks but it's all the Network Extension framework. I don't see any of the System Configuration framework in there.

So, while I hope it's a trivial change to implement it, I'm not even sure where to begin 😅

[phatmandrake]

Does this mean moving everything over to the Network Extension framework or is it possible just to invoke part of it 😅?

As a workaround for now I'm just forcibly setting the service order on the network interfaces to prioritize the VPN connection when it's active.

[halo]

I'm sorry, I didn't read your previous posts thoroughly enough. I understand your problem a bit better now and would ask you to help me one more time. Specifically, I need help to reproduce the problem and verify whether it is solved.

1. How can I programmatically (terminal) check whether "Route all traffic over VPN" option is actually working or not?
   I'm under the impression that macosvpn does not activate that setting properly, but using the GUI does (i.e. toggling it off and on in the network settings will actually turn it on). I want to rule out that problem.

2. searchdomains can be configured using the old SystemConfiguration framework macosvpn uses.
   How do you use networksetup to add a searchdomain and how do you verify in scutil --dns that it works?

Does this mean moving everything over to the Network Extension framework or is it possible just to invoke part of it 😅?

NEF does not support L2TP, so we cannot use it. I hope that the searchdomain thingy will solve the problem rather than going down the NEF rabbit hole :)

As a workaround for now I'm just forcibly setting the service order on the network interfaces to prioritize the VPN connection when it's active.

Do you do that manually or programmatically? I assume the SystemPreferences framework has the capability to rearrange the order of services, too.