Minikube VM inherits macOS DNS-over-TLS proxy (srp-mdns-proxy on port 853), causing DNS resolution failures inside VM

[outcoldman]

What Happened?

I’m experiencing consistent DNS resolution failures inside a Minikube VM on macOS, and after debugging I traced the issue to macOS exposing a DNS-over-TLS proxy (srp-mdns-proxy) on port 853, which Minikube appears to inherit as its upstream resolver.

The issue started when I noticed the standard, which is being referenced in some other issues, but those were Windows (which also maybe related to it)

Failing to connect to https://registry.k8s.io/ from inside the minikube VM

After that I tried once to create a ha cluster with --ha --nodes=6 and that is where I have noticed issues with DNS.

I have two Macs, one does not have this issue, the other one does. I have found possible a reason, why one of them fail, it is this process on macOS

sudo lsof -i :853

COMMAND   PID           USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
srp-mdns- 778 _mdnsresponder    5u  IPv6 0xba1ae837bbad06b4      0t0  TCP *:domain-s (LISTEN)

Both Macs have the same macOS version (26.4), one is MBP, another one is Mac Pro. So I guess that might be a difference, why one has this process, and another one does not.

The issue is, not only Failing to connect to https://registry.k8s.io/ from inside the minikube VM, but also

curl https://google.com
curl: (6) Could not resolve host: google.com

But!

$ nslookup google.com 1.1.1.1
Server:		1.1.1.1
Address:	1.1.1.1:53

Non-authoritative answer:
Name:	google.com
Address: 2607:f8b0:4008:809::200e

Non-authoritative answer:
Name:	google.com
Address: 142.251.35.238

And

resolvectl status

Global
           Protocols: +LLMNR +mDNS DNSOverTLS=opportunistic DNSSEC=no/unsupported
    resolv.conf mode: uplink
Fallback DNS Servers: 1.1.1.1#cloudflare-dns.com 8.8.8.8#dns.google 1.0.0.1#cloudflare-dns.com 8.8.4.4#dns.google 2606:4700:4700::1111#cloudflare-dns.com 2001:4860:4860::8888#dns.google
                      2606:4700:4700::1001#cloudflare-dns.com 2001:4860:4860::8844#dns.google

Link 2 (eth0)
    Current Scopes: DNS LLMNR/IPv4
         Protocols: +DefaultRoute +LLMNR -mDNS DNSOverTLS=opportunistic DNSSEC=no/unsupported
Current DNS Server: 192.168.65.1
       DNS Servers: 192.168.65.1

On the host, if I run

sudo netstat -anv | grep 853

I get a LOT of

tcp4       0      0  192.168.65.1.853       192.168.65.24.49286    TIME_WAIT           1124          845  131392  131768   srp-mdns-proxy:778    02131 00010004 0000000000020149 00000081 01080800      0      0 000000

And that is where I luck some knowledge what is actually causing this issue. Is that srp-mdns-proxy failing, or is it minikube does not talk correctly to it. Maybe DNSoverTLS should be disabled?

I will be happy to provide more details, if required. And going to dig more to see if I can find some information.

Attach the log file

See description

Operating System

macOS (Default)

Driver

vfkit

[outcoldman]

Just a quick fix after I started the VM

sudo mkdir -p /etc/systemd/resolved.conf.d
sudo vi /etc/systemd/resolved.conf.d/no-dot.conf

with content

[Resolve]
DNSOverTLS=no

sudo systemctl restart systemd-resolved

So I guess... Maybe that can be an option to place that in the ISO...

[outcoldman]

I made a private build that writes that inside the VM, so minikube will have an option for me --dns-over-tls=false which solved this problem for me.

I am happy to provide a PR, just not sure if that is the route you would want to take, or just place the DNSOverTLS=no inside of the ISO by default. I tried that first, but Docker for Mac crashed on my Mac, while building ISO. So to unblock myself went that route of patching minikube

Also to think about it, maybe will be better to just implement some kind of option for user to specify any command on start, like cloud-init. Considering minikube already syncing files, maybe just add something like

minikube start --user-init=/my-custom-script.sh

That way if user wants to add some custom configurations to the instance, or install something additional, they will be able to do that. And in my case that would allow to "fix" things, like to disable DNSOverTLS

[afbjorklund]

If you add a script called /var/lib/boot2docker/bootlocal.sh, it will be called during provision.

Minikube doesn't use cloud-init, but an older variant based on "user-data.tar" in the disk image.

[outcoldman]

@afbjorklund thank you for the suggestion, there is not a lot of documentation about this file, and I tried to place it as ~/.minikube/files/var/lib/boot2docker/bootlocal.sh with content

#!/bin/sh
mkdir -p /etc/systemd/resolved.conf.d
printf '[Resolve]\nDNSOverTLS=no\n' \
  > /etc/systemd/resolved.conf.d/no-dns-over-tls.conf
systemctl restart systemd-resolved

That did not do anything for me.

And when I ssh to minikube, that file does not exist under /var/lib/boot2docker/

So maybe the code that should invoke /var/lib/boot2docker/bootlocal.sh exists, but there is no easy way to actually sync file there.

[afbjorklund]

The script runs post-start, that is after Docker is up and running. It used to be documented in Docker Machine (only)

[outcoldman]

@afbjorklund right, but the issue it, how can I place it there?

[afbjorklund]

Hmm, it should have ended up there next to the "user.tar"...

And there should be traces of it in the minikube log file, as well.

I0330 13:48:01.831740   62368 filesync.go:126] Scanning /home/anders/.minikube/files for local assets ...
I0330 13:48:01.831807   62368 filesync.go:149] local asset: /home/anders/.minikube/files/var/lib/boot2docker/bootlocal.sh -> bootlocal.sh in /var/lib/boot2docker
I0330 13:48:01.831849   62368 ssh_runner.go:195] Run: sudo mkdir -p /var/lib/boot2docker
I0330 13:48:01.841945   62368 ssh_runner.go:362] scp /home/anders/.minikube/files/var/lib/boot2docker/bootlocal.sh --> /var/lib/boot2docker/bootlocal.sh (33 bytes)

Main problem would be that it is "too late" for the initial boot

[outcoldman]

Yeah, it does it (I have $MINIKUBE_HOME pointed to external volume, just FYI)

scp /Volumes/DATA/minikube/.minikube/files/var/lib/boot2docker/bootlocal.sh --> /var/lib/boot2docker/bootlocal.sh (174 bytes)

And, I tried to play with that file a bit, updated to make sure permissions are right

#!/bin/sh

sudo touch /var/lib/boot2docker/bootlocal.test

echo "bootlocal.sh started" | sudo tee /var/log/bootlocal.log > /dev/null

# Disable DNS over TLS globally
sudo mkdir -p /etc/systemd/resolved.conf.d
printf '[Resolve]\nDNSOverTLS=no\n' \
  | sudo tee /etc/systemd/resolved.conf.d/no-dns-over-tls.conf > /dev/null
sudo systemctl restart systemd-resolved

echo "bootlocal.sh executed" | sudo tee /var/log/bootlocal.log > /dev/null

I just don't see it being executed, no /var/lib/boot2docker/bootlocal.test or no /var/log/bootlocal.log created. But if I try to ssh to it after - I can see that I can execute it.

Curious if this https://github.com/kubernetes/minikube/blob/master/deploy/iso/minikube-iso/package/automount/minikube-automount#L218 happens before scp command, so that is why it never gets executed...

[afbjorklund]

Curious if this minikube-automount happens before scp command, so that is why it never gets executed...

Only on the first boot, on the second boot the file should already be there so it should be executed during mount.

In the original version, it was executed by init.

• Bring back bootlocal.sh #1111

[outcoldman]

@afbjorklund gotcha, but in this case bootlocal.sh cannot really help me, as in case if I want to start minikube with --ha --nodes=6 will fail on boot, because of the DNS issue. So I guess minikube does need some ability to invoke similar to bootlocal.sh on boot, or have ability to disable DNSoverTLS - but this is just my case.

[nirs]

@outcoldman this smells like #22646 - your broken Mac is managed?

Minikube seems to use the host as DNS server:

Current DNS Server: 192.168.65.1
       DNS Servers: 192.168.65.1

I guess you use vfkit driver without --network vmnet-shared?

Replace the dns server to public DNS servers (e.g. 8.8.8.8 1.1.1.1) and DNS may work again.

[outcoldman]

@nirs funny thing is that the "working" macOS is managed, but "broken" is not. I have seen #22646, but not sure this is accurate in my case:

On managed Macs, network extensions block DNS traffic from the minikube VM to the host's DNS resolver (mDNSResponder), but DNS traffic to public DNS servers passes.

What I do see on the "broken" Mac, is that I do have that _mdnsresponder running on port 853. And on managed/working Mac I do not have this process. My guess is HomeKit and because this MacPro is stationary - makes macOS to run it, to maybe run some additional services.

Also my guess is that minikube VM does try to use 853 for DNS queries, but there is some mismatch for the protocol. If I run sudo netstat -anv | grep 853 on macOS I see multiple connections with the same (over hundreds of them)

tcp4       0      0  192.168.65.1.853       192.168.65.32.53348    TIME_WAIT           1124          844  131392  131768   srp-mdns-proxy:1159   02131 00010004 00000000010661c3 00000081 01080800      0      0 000000

After I disable DNSoverTCP I don't have any problems anymore.

And in my case nslookup works, but DNS resolution over resolver does not, because nslookup just uses port 53 and resolver uses 853

$ curl google.com
curl: (6) Could not resolve host: google.com
$ nslookup google.com
Server:		192.168.65.1
Address:	192.168.65.1:53

Non-authoritative answer:
Name:	google.com
Address: 142.251.34.142

Non-authoritative answer:
Name:	google.com
Address: 2607:f8b0:4008:802::200e

Right now I am just disabling DNSoverTCP in minikube - and everything works. And I do need local DNS resolution.

[nirs]

@outcoldman Consider updating the docs about this issue. Since this was found with vfkit, maybe the troubleshooting section can show how to disable DNSOverTCP?

https://minikube.sigs.k8s.io/docs/drivers/vfkit/#troubleshooting

[outcoldman]

Considering that I need to disable it on boot (specifically for the case trying to bootstrap -ha -nodes 6 cluster), the only solution I have found is to patch the minikube binary with code below (as bootlocal.sh does not help me, and disabling it later does not solve the problem with -ha -nodes 6)

diff --git a/pkg/provision/provision.go b/pkg/provision/provision.go
index 81767e116..846d50d70 100644
--- a/pkg/provision/provision.go
+++ b/pkg/provision/provision.go
@@ -225,12 +225,33 @@ func setRemoteAuthOptions(p provision.Provisioner) auth.Options {
        return authOptions
 }
 
+// configureSystemdResolved writes a drop-in config to disable DNS-over-TLS in
+// systemd-resolved, which can interfere with DNS resolution inside the VM.
+func configureSystemdResolved(p miniProvisioner) error {
+       const dst = "/etc/systemd/resolved.conf.d/no-dns-over-tls.conf"
+       const content = "[Resolve]\nDNSOverTLS=no\n"
+       klog.Infof("configuring systemd-resolved: writing %s", dst)
+       if _, err := p.SSHCommand(fmt.Sprintf(
+               "sudo mkdir -p %s && printf %%s \"%s\" | sudo tee %s && sudo systemctl restart systemd-resolved",
+               path.Dir(dst), content, dst,
+       )); err != nil {
+               return err
+       }
+       return nil
+}
+
 func setContainerRuntimeOptions(name string, p miniProvisioner) error {
        c, err := config.Load(name)
        if err != nil {
                return fmt.Errorf("getting cluster config: %w", err)
        }
 
+       if !c.DNSOverTLS {
+               if err := configureSystemdResolved(p); err != nil {
+                       klog.Warningf("configureSystemdResolved failed: %v", err)
+               }
+       }
+
        switch c.KubernetesConfig.ContainerRuntime {
        case "crio", "cri-o":
                return setCrioOptions(p)

I obviously be happy to contribute this patch, just based on the #22711 (comment) - it seems like you are all overwhelmed with PR right now, don't want to create one more hanging.

[nirs]

I need to configure DNS for #22646 right after starting the vm but had no time to work on this yet. If you can create a clean way to configure systemd-resolved that we can extend later this can be very helpful.

I think that /etc is not persisted in minikube, so when a note boot you are using again the /etc from the ISO, so configuring dynamiclaly using resolvectl seems to be the right way.

Example used for fixing #22646:

minikube ssh sudo bash -c "
cat > /etc/systemd/network/00-static-dns.network <<EOF
[Match]
Name=eth*
[Network]
DHCP=yes
DNS=8.8.8.8 1.1.1.1
[DHCP]
UseDNS=false
EOF
resolvectl dns eth0 8.8.8.8 1.1.1.1
resolvectl domain eth0 "~."
resolvectl flush-caches
"