DNS-resolving on Node with coredns don't work

[tobiasf88]

What did you do

• How was the cluster created?

# yaml-language-server: $schema=https://raw.githubusercontent.com/rancher/k3d/main/pkg/config/v1alpha3/schema.json
apiVersion: "k3d.io/v1alpha3"
kind: Simple
name: devcluster
servers: 1
agents: 1
kubeAPI:
 host: "127.0.0.1"
 hostPort: "6443"
 hostIP: "127.0.0.1"
ports:
 - port: 80:80
   nodeFilters:
     - loadbalancer
 - port: 443:443
   nodeFilters:
     - loadbalancer
options:
 k3s:
   extraArgs:
     - arg: --no-deploy=traefik
       nodeFilters:
         - server:*

• What did you do afterwards?

kubectl apply -f YAML-FILES

I have 2 Settings:

One on my Mac with "Docker for Mac": All fine
One on a Linux Machine: Pods on the same Server/Agent with coredns don't get the coredns response.

What did you expect to happen

All works fine

Investigation

I check the dns-resolving with

nslookup SERVICE_NAME

this will work on all node, where no dnscore pod is in. On the node with the coredns pod, die pods can't communicate with each other (nslookup fails). Direct IP-Connection is working.

For that, I increase logging of coredns. And now, it will be crazy. The coredns received all dns query's and resolved it - NOERROR. But the response don't received the pod which asked. In the Pod I get a timeout.

But, same Configs, same Application on my Mac --> all works fine. I have no idea whats the problem on the Linux-Server

Which version of k3d

• output of k3d version

Which version of docker

docker version

Client: Docker Engine - Community
 Version:           20.10.9
 API version:       1.41
 Go version:        go1.16.8
 Git commit:        c2ea9bc
 Built:             Mon Oct  4 16:08:14 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.9
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.8
  Git commit:       79ea9d3
  Built:            Mon Oct  4 16:06:37 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.9
  GitCommit:        e25210fe30a0a703442421b0f60afac609f950a3
 runc:
  Version:          1.0.1
  GitCommit:        v1.0.1-0-g4144b63
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.6.3-docker)
  scan: Docker Scan (Docker Inc., v0.9.0)

Server:
 Containers: 5
  Running: 5
  Paused: 0
  Stopped: 0
 Images: 7
 Server Version: 20.10.9
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e25210fe30a0a703442421b0f60afac609f950a3
 runc version: v1.0.1-0-g4144b63
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 3.10.0-1160.45.1.el7.x86_64
 Operating System: Red Hat Enterprise Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 48
 Total Memory: 125.4GiB
 Name: XXXXXXX
 ID: XXXXXX
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  XXXXXXXXXXX
 Live Restore Enabled: false

[iwilltry42]

Hi @tobiasf88 , thanks for starting this discussion!
The strangest part to me is that it's failing on the same node (container).. 🤔
Do you have ufw or firewalld or something the like running on your RHEL server? (see related comments here and here)
Not sure if it could be related to what's mentioned in those K3s docs (also about firewalld) 🤔

[tobiasf88]

✅ solved --> restart the server solved all problems 🤦🏻‍♂️

[iwilltry42]

Well.. I like that solution 😂 👍

[tobiasf88]

PR --> Readme.md if you have any strange problems, restart the Server! ?? 😅

[iwilltry42]

"Have you tried turning it off and on again?" in the Issue Template? 😁