DNS resolution broken on IPv6-only clusters with Docker: enable_network_magic() is IPv4-only

[matejvasek]

Title: DNS resolution broken on IPv6-only clusters with Docker: enable_network_magic() is IPv4-only

What happened:

On an IPv6-only cluster (ipFamily: ipv6) running on Docker, CoreDNS pods cannot resolve external names. The node's /etc/resolv.conf contains nameserver 172.18.0.1 (the IPv4 default gateway), which is unreachable from IPv6-only pods. CoreDNS logs show:

[ERROR] plugin/errors: 2 ghcr.io. AAAA: read udp [fd00:10:244::43]:59624->[172.18.0.1]:53: connect: network is unreachable

The root cause is in enable_network_magic() in the node entrypoint script. It replaces Docker's embedded DNS (127.0.0.11) with the IPv4 default gateway:

docker_host_ip=$(ip -4 route show default | cut -d' ' -f3)

This is hardcoded to IPv4. On an IPv6-only cluster, the resulting nameserver (172.18.0.1) is unreachable from pods that only have IPv6 addresses.

Note: even if the entrypoint used ip -6 route show default to get the IPv6 gateway (fc00:f853:ccd:e793::1), DNS still wouldn't work because Docker's DNS proxy does not listen on the IPv6 bridge gateway address (see moby/moby#41651). So the entrypoint fix alone is not sufficient — but it is a prerequisite for any future Docker fix to work.

What you expected to happen:

CoreDNS pods on an IPv6-only cluster should be able to forward DNS queries to an upstream resolver. The entrypoint should configure at least one IPv6-reachable nameserver in /etc/resolv.conf when the cluster uses ipFamily: ipv6.

How to reproduce it (as minimally and precisely as possible):

cat <<EOF | kind create cluster --name test --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  ipFamily: ipv6
EOF

# Check resolv.conf — only has IPv4 nameserver
docker exec test-control-plane cat /etc/resolv.conf
# nameserver 172.18.0.1

# The original Docker resolv.conf had 127.0.0.11
docker exec test-control-plane cat /etc/resolv.conf.original
# nameserver 127.0.0.11

# CoreDNS pods can't reach the IPv4 nameserver — verify from a debug pod
kubectl run -it --rm debug --image=alpine --restart=Never -- nslookup google.com
# ;; connection timed out; no servers could be reached

kind delete cluster --name test

Anything else we need to know?:

The entrypoint's enable_network_magic() function is entirely IPv4-centric:

• Uses ip -4 route show default to find the host IP
• Only patches iptables (not ip6tables)
• Only resolves host.docker.internal via getent ahostsv4

On Podman this works by accident: Podman's aardvark-dns puts its own IPv6 nameserver in /etc/resolv.conf (not 127.0.0.11), so the sed replacement is a no-op and the working IPv6 nameserver survives.

There is also a Docker-side issue: Docker's DNS proxy only listens on the IPv4 bridge gateway, not the IPv6 one (moby/moby#41651, open since 2020). Even if kind fixed the entrypoint to use the IPv6 gateway, there's currently nothing listening on port 53 there. A complete fix requires both kind and Docker changes.

As a workaround, running a socat DNS proxy on the node bridges the gap:

# Get the node's IPv6 address
NODE_IPV6=$(docker inspect test-control-plane -f '{{.NetworkSettings.Networks.kind.GlobalIPv6Address}}')

# Install socat and start DNS proxies (UDP + TCP)
docker exec test-control-plane bash -c "
  apt-get update -qq >/dev/null 2>&1 && apt-get install -y -qq socat >/dev/null 2>&1
  socat UDP6-LISTEN:53,bind=[${NODE_IPV6}],fork,reuseaddr UDP4:172.18.0.1:53 &
  socat TCP6-LISTEN:53,bind=[${NODE_IPV6}],fork,reuseaddr TCP4:172.18.0.1:53 &
"

# Patch CoreDNS to forward to the node's IPv6 address instead of /etc/resolv.conf
kubectl get cm coredns -n kube-system -o json \
  | sed "s|forward . /etc/resolv.conf|forward . ${NODE_IPV6}|" \
  | kubectl apply -f -
kubectl rollout restart deployment/coredns -n kube-system

Environment:

• kind version: v0.31.0 (also reproduced with v0.29.0)
• Runtime info: Docker 29.3.0
• Kubernetes version: v1.34.0

[matejvasek]

possibly related: #3114

[BenTheElder]

There is also a Docker-side issue: Docker's DNS proxy only listens on the IPv4 bridge gateway, not the IPv6 one (moby/moby#41651, open since 2020). Even if kind fixed the entrypoint to use the IPv6 gateway, there's currently nothing listening on port 53 there. A complete fix requires both kind and Docker changes.

Docker has previously stated they won't do this. So to support pure-single stack we'll have to do something like the socat bridge.

cc @aojea we discussed this before but I don't think a commitment to a particular solution materialized.

I think generally we will need to run some kind of ideally very minimal proxy on the nodes and also leverage it to drop the iptables hackery.

[matejvasek]

I can understand that they do not wnat run DNS resolver on the loopback (::1) address.

But it does not run even on the Gateway (i.e. some ULA address typically).

[BenTheElder]

Yeah, unfortunately it didn't seem like the project has any plans for this moby/moby#41651 (comment)

And even if they do, they'd like us to not depend on the iptables bits (understandable ... though we would probably just use our own iptables/nftables rules if it was a normal predictable socket).

I'm open to fixing this on the kind end by running a proxy service on the node, I'm just concerned about keeping it maintainable (bugs, security fixes) and cheap (complexity, cpu/memory, latency, image size) and a little buried at the moment. Looking for @aojea's thoughts as well.

It will be a bit ridiculous going from coreDNS => kind-node-proxy => docker socket / dockerd => host resolver but ... oh well, it already is a bit.

[matejvasek]

I just checked now with a testing container and it appears that moby does not serve DNS at the gateway at all. Strange I would swear it was there before at least on IPv4 gateway IP.

[matejvasek]

But in the KinD control plane container it works (querying gateway IP (e.g. 172.18.0.1)). Is that KinD modification?

[matejvasek]

The proxy could be simple Go program probably. If size is of essence it could be part of an existing binary as sub-command or as multi-call (dispatch on arg[0]).

[matejvasek]

But in the KinD control plane container it works (querying gateway IP (e.g. 172.18.0.1)). Is that KinD modification?

Oh that's probably in the iptables magic.

[matejvasek]

It was little bit confusing because podman does in fact serve DNS on the gateway IPs. Apparently moby does not do that at all.

[aojea]

I'm open to fixing this on the kind end by running a proxy service on the node, I'm just concerned about keeping it maintainable (bugs, security fixes) and cheap (complexity, cpu/memory, latency, image size) and a little buried at the moment. Looking for @aojea's thoughts as well.

I started that branch some time ago but stopped, there were certain things I didn't have time to followup:

• route_localnet=1 only works for IPv4, so local generated packets are able to be DNATed to localhost addresses in IPv4 but not IPb6, but this can be overcome in a different way, some thoughts:
  • listen in the eth0 address
  • adding a well known link local to the loopback interface and listen there
  • bind to a non local address https://oswalt.dev/2022/02/non-local-address-binds-in-linux/
  • use tproxy
  • ...
• detection of IPv6 ONLY clusters during node startup, so we put the right address on resolv.conf
  

  kind/images/base/files/usr/local/bin/entrypoint

  Lines 446 to 456 in 71f3111

  	enable_network_magic(){
  	# well-known docker embedded DNS is at 127.0.0.11:53
  	local docker_embedded_dns_ip='127.0.0.11'
  	# first we need to detect an IP to use for reaching the docker host
  	local docker_host_ip
  	docker_host_ip="$( (head -n1 <(timeout 5 getent ahostsv4 'host.docker.internal') | cut -d' ' -f1) || true)"
  	# if the ip doesn't exist or is a loopback address use the default gateway
  	if [[ -z "${docker_host_ip}" ]] || [[ $docker_host_ip =~ ^127\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
  	docker_host_ip=$(ip -4 route show default | cut -d' ' -f3)
  	fi

The proxy thing is simple, I asked copilot to clean my code and publish as a standalone proxy in https://github.com/aojea/dnsproxy64 , this is the same as

  socat UDP6-LISTEN:53,bind=[${NODE_IPV6}],fork,reuseaddr UDP4:172.18.0.1:53 &
  socat TCP6-LISTEN:53,bind=[${NODE_IPV6}],fork,reuseaddr TCP4:172.18.0.1:53 &

with the difference that the udp proxy is aware of the UDP/DNS session so it can track request/responses. We can also modify the golang code to open the listening sockets with IP_TRANSPARENT or IP_FREEBIND, I already have this working in kindnet for a long time for the same purpose https://github.com/search?q=repo%3Akubernetes-sigs%2Fkindnet+IP_TRANSPARENT&type=code .

We can ship the userspace proxy and inject in the image to start with a systemd unit per example.

The proxy can do "more magic" and also use the systemd socket activation directly https://github.com/coreos/go-systemd#socket-activation and scan the local sockets to find the docker embedded proxy, that will leave us with the detection of IPv6 only clusters and the configuration of the systemd socket unit.