Getting the "real IP" from a request inside of a k3d cluster

[dreimsbach]

I run a k3d cluster of one node on a Debian 11 server. I've disabled Traefik and use nginx as Ingress controller.
I have several different applications running in the cluster that are accessible via subdomains of my main domain eg. app1.my-domain.com, app2.my-domain.com, ...

Originally I wasn't able to get the original IP from requests on those apps. X-Forwarded-For was always the IP of the docker bridge. I really searched a lot for solutions. Only thin that helped was using the host network. Then it was fine and I am getting those real IPs.

With updating to K3d 5.03, there is a bug preventing the use of host networks. Anyway, it's a workaround and I like to get rid of it as I really want to have more nodes in the cluster.

That's why I finally ask here for help. Thats my cluster creation with k3d 4.4.8 (when host network was working properly).

k3d cluster create my-cluster \
  --port 8089:8089@loadbalancer  \
  --port 80:80@loadbalancer  \
  --port 443:443@loadbalancer  \
  --servers 1  \
  --volume "xxx:xxx/" \
  --volume "xxx/base/helm/helm-ingress-nginx.yaml:/var/lib/rancher/k3s/server/manifests/helm-ingress-nginx.yaml" \
  --k3s-server-arg '--no-deploy=traefik' \
  --servers-memory=18g \
  --network host \
  --api-port 1.2.3.4:6443 \
  --k3s-server-arg '--advertise-address=1.2.3.4' \
  --k3s-server-arg '--tls-san=1.2.3.4'

My nginx ingress looks like this

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: main-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      ## some things
    nginx.ingress.kubernetes.io/server-snippet: |
      ## some things
spec:
  rules:
  - host: "app1.my-domain.com"
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: app1
            port:
              number: 80
  - host: "app2.my-domain.com"
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: app2
            port:
              number: 80

With that configuration I get the real IP because I use the "host" network. When I remove that arg from creating the cluster, the IPs I get are the one from the docker bridge.

What do I wrong here?

[iwilltry42]

Hi @dreimsbach , thanks for starting this discussion!

First off: What's the bug you faced when using hostnetwork? From the command you pasted there, I assume that the problem is with the loadbalancer, which shouldn't be there (and doesn't make sense) in the hostnetwork setup.

There seems to be an open issue on this for ages (also applies for Docker on Linux) which is also still open on the roadmap.
On Linux there seems to be a workaround as per this comment.
Others report that disabling the userland-proxy makes this work as well (see docker documentation on how to do this).