Refactor aro-dnsmasq-pre.sh to not overwrite /etc/resolv.conf

[ventifus]

Which issue this PR addresses:

Fixes ARO-15180
Companion installer PR openshift/installer-aro-wrapper#255

Derived from the method in https://github.com/openshift/machine-config-operator/blob/master/templates/common/gcp/files/usr-local-bin-update-dns-server.yaml

What this PR does / why we need it:

We've been overwriting /etc/resolv.conf. NetworkManager owns this file and if NetworkManager needs to refresh it we will lose our changes. Instead, create a NetworkManager drop-in /etc/NetworkManager/conf.d/dns-servers.conf with the node's IP.

Test plan for issue:

Is there any documentation that needs to be updated for this PR?

No, but the change needs to be socialized amongst ARO SRE since it affects how nameservers are managed.

How do you know this will function as expected in production?

Testing has been done with an extant UDR+bad dns cluster to ensure there are no external DNS dependencies. Nodes boot and scale correctly.

[ventifus]

/azp run ci,e2e

[azure-pipelines]

Azure Pipelines successfully started running 2 pipeline(s).

[kimorris27]

What testing do you think we should do before this merges?

[tsatam]

Do we need to make a corresponding change to what the installer puts down? My understanding is that for new clusters, the changes in our operator won't get applied until the cluster is first upgraded, and the cluster will run with what the installer has until then.

[hawkowl]

@tsatam When the Operator is first installed, it is set to allow all reconciliations, and then that is switched to only on upgrades at the end of the install process. So, this will apply to new clusters (at the cost of a reboot + install time, so we should also update the installer wrapper).

[ventifus]

I've tested this now in a UDR + misconfigured DNS cluster (vnet dns = 172.16.0.0). I set aro.dnsmasq.enabled: "false" and manually edited 99-master-aro-dns and 99-worker-aro-dns to have the new content.

After all nodes roll out they have the following config, which looks good

sh-5.1# cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 10.0.0.7

sh-5.1# cat /etc/resolv.conf.dnsmasq
# Generated by NetworkManager
search reddog.microsoft.com
nameserver 172.16.0.0

sh-5.1# cat /etc/NetworkManager/conf.d/dns-servers.conf
# Added by dnsmasq.service
[global-dns-domain-*]
servers=10.0.0.7

I made sure all the cluster operators were healthy, and worker machinesets can scale up.

N.B. even with this change we still end up touching /etc/resolv.conf with dnsmasq.service's

ExecStopPost=/bin/bash -c '/bin/mv /etc/resolv.conf.dnsmasq /etc/resolv.conf; /usr/sbin/restorecon /etc/resolv.conf'

I'll fix that up to delete dns-servers.conf instead.

[ventifus]

A concern I have with this is we're losing search. Depending on pod configuration, this can be exposed to user workloads so they may depend on it. I'll see if that's something we can preserve.

[ventifus]

Ok I've fixed the search domain by adding a [global-dns] section.

sh-5.1# cat /etc/resolv.conf
# Generated by NetworkManager
search reddog.microsoft.com
nameserver 10.0.2.7

sh-5.1# cat /etc/resolv.conf.dnsmasq
# Generated by NetworkManager
search reddog.microsoft.com
nameserver 172.16.0.0

sh-5.1# cat /etc/NetworkManager/conf.d/dns-servers.conf
# Added by dnsmasq.service
[global-dns]
searches=reddog.microsoft.com

[global-dns-domain-*]
servers=10.0.2.7

[ventifus]

Companion installer PR openshift/installer-aro-wrapper#255

[zaneb]

Unless you were having trouble with the code to retrieve the search domains and IP addresses, I'd be tempted to keep it the same.

[ventifus]

We do have problems with the existing code. It tries to make guesses about which network interface to use based on if the interface br-ex exists. We've seen a number of instances where this fails, particularly if the service startup order changes.

[ventifus]

@zaneb Another approach to finding the search domain

for DEV in $(nmcli --fields device,state,type --terse device | awk 'BEGIN {FS=":"} ; {if ($2 == "connected") { print $1 }}'); do nmcli dev sho $DEV | awk 'BEGIN {FS=":\\s*"}; { if ($1 ~ /DOMAIN/ && $2 ~ /.+/) { print $2} }'; done | sort -u

[zaneb]

Fair enough, non-determinism is definitely not what you want here 😄

Looping over all interfaces doesn't look that bad though.