What happened: HTTP returns content from a wrong backend

What you expected to happen: HTTP returns content from a correct backend which the ingress is configured.

What do you think went wrong?:

links to #11937

func upstreamName outputs the same upstream name for different ingress backend.

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):
v1.11.1 (and the latest main)

Kubernetes version (use kubectl version):

Environment:

• Cloud provider or hardware configuration: local

• OS (e.g. from /etc/os-release): wsl ubuntu

• Kernel (e.g. uname -a):

• Install tools:

  • Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc.

• Basic cluster related info:Linux 5.15.146.1-microsoft-standard-WSL2 Basic structure  #1 SMP Thu Jan 11 04:09:03 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

  • kubectl version

  Client Version: v1.30.0
  Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
  Server Version: v1.29.2
  

  • kubectl get nodes -o wide

  NAME                 STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION                       CONTAINER-RUNTIME
  kind-control-plane   Ready    control-plane   10d   v1.29.2   172.19.0.2    <none>        Debian GNU/Linux 12 (bookworm)   5.15.146.1-microsoft-standard-WSL2   containerd://1.7.13
  

• How was the ingress-nginx-controller installed:

  • If helm was used then please show output of helm ls -A | grep -i ingress

  nginx           ingress-nginx           3               2024-09-06 18:44:48.292292472 +0800 HKT deployed        ingress-nginx-4.11.1    1.11.1
  

  • If helm was used then please show output of helm -n <ingresscontrollernamespace> get values <helmreleasename>

  USER-SUPPLIED VALUES:
  controller:
    admissionWebhooks:
      certManager:
        enabled: true
      patch:
        image:
          digest: ""
          registry: registry.local:5000
    extraArgs:
      publish-status-address: 127.0.0.1
    hostPort:
      enabled: true
    image:
      digest: ""
      digestChroot: ""
      registry: registry.local:5000
    ingressClassResource:
      default: true
    metrics:
      enabled: true
      port: 10254
    nodeSelector:
      ingress-ready: "true"
      kubernetes.io/os: linux
    publishService:
      enabled: false
    updateStrategy:
      rollingUpdate:
        maxUnavailable: 1
  

• Current State of the controller:

  • kubectl describe ingressclasses

  Name:         nginx
  Labels:       app.kubernetes.io/component=controller
                app.kubernetes.io/instance=nginx
                app.kubernetes.io/managed-by=Helm
                app.kubernetes.io/name=ingress-nginx
                app.kubernetes.io/part-of=ingress-nginx
                app.kubernetes.io/version=1.11.1
                helm.sh/chart=ingress-nginx-4.11.1
  Annotations:  ingressclass.kubernetes.io/is-default-class: true
                meta.helm.sh/release-name: nginx
                meta.helm.sh/release-namespace: ingress-nginx
  Controller:   k8s.io/ingress-nginx
  Events:       <none>
  

  • kubectl -n <ingresscontrollernamespace> get all -o wide
  • kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>
  • kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>

• Current state of ingress object, if applicable:

  • kubectl -n <appnamespace> get all,ing -o wide

  NAME                                                  READY   STATUS    RESTARTS      AGE   IP            NODE                 NOMINATED NODE   READINESS GATES
  pod/nginx-ingress-nginx-controller-76dcf989d8-v2qm8   1/1     Running   2 (24h ago)   10d   10.244.0.14   kind-control-plane   <none>           <none>
  
  NAME                                               TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE   SELECTOR
  service/nginx-ingress-nginx-controller             LoadBalancer   10.96.75.220    <pending>     80:31222/TCP,443:31220/TCP   10d   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx,app.kubernetes.io/name=ingress-nginx
  service/nginx-ingress-nginx-controller-admission   ClusterIP      10.96.138.105   <none>        443/TCP                      10d   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx,app.kubernetes.io/name=ingress-nginx
  service/nginx-ingress-nginx-controller-metrics     ClusterIP      10.96.152.203   <none>        10254/TCP                    10d   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx,app.kubernetes.io/name=ingress-nginx
  
  NAME                                             READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                                                 SELECTOR
  deployment.apps/nginx-ingress-nginx-controller   1/1     1            1           10d   controller   registry.local:5000/ingress-nginx/controller:v1.11.1   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx,app.kubernetes.io/name=ingress-nginx
  
  NAME                                                        DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES                                                 SELECTOR
  replicaset.apps/nginx-ingress-nginx-controller-76dcf989d8   1         1         1       10d   controller   registry.local:5000/ingress-nginx/controller:v1.11.1   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=76dcf989d8
  

  • kubectl -n <appnamespace> describe ing <ingressname>
  • If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag

• Others:
  The full manifests that reproduce this problem:

# pod1
apiVersion: v1
kind: Pod
metadata:
  name: "pod1"
  labels:
    app: "pod1"
spec:
  containers:
    - name: pod1
      image: "busybox:latest"
      command:
        - "sh"
        - "-c"
        - |
          while true; do
          echo -e "HTTP/1.1 200 OK\n\n Responsing From: $HOSTNAME" | nc -l -p 8000;
          done;
      resources:
        limits:
          cpu: 200m
          memory: 500Mi
        requests:
          cpu: 100m
          memory: 200Mi
      ports:
        - containerPort: 8000
---
# pod2
apiVersion: v1
kind: Pod
metadata:
  name: "pod2"
  labels:
    app: "pod2"
spec:
  containers:
    - name: pod1
      image: "busybox:latest"
      command:
        - "sh"
        - "-c"
        - |
          while true; do
          echo -e "HTTP/1.1 200 OK\n\n Responsing From: $HOSTNAME" | nc -l -p 8000;
          done;
      resources:
        limits:
          cpu: 200m
          memory: 500Mi
        requests:
          cpu: 100m
          memory: 200Mi
      ports:
        - containerPort: 8000
---
# service1
apiVersion: v1
kind: Service
metadata:
  name: "service-pod"
spec:
  selector:
    app: "pod1"
  ports:
    - protocol: "TCP"
      port: 80
      targetPort: 8000
      name: http
---
# service2
apiVersion: v1
kind: Service
metadata:
  name: "service"
spec:
  selector:
    app: "pod2"
  ports:
    - protocol: "TCP"
      port: 80
      targetPort: 8000
      name: pod-http
---
# ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress
spec:
  rules:
    - host: "example.local"
      http:
        paths:
          - path: "/service1"
            pathType: Prefix
            backend:
              service:
                name: "service-pod"
                port:
                  name: "http"
          - path: "/service2"
            pathType: Prefix
            backend:
              service:
                name: "service"
                port:
                  name: "pod-http"

How to reproduce this issue:

$ curl example.local/service1
 Responsing From: pod1

$ curl example.local/service2
 Responsing From: pod1
# expected: Responsing From: pod2