Skip to content

Nginx Controller does not update configuration while recreating service. #11963

New issue
Jump to bottom
New issue
Jump to bottom
Open
#12034
Open
Nginx Controller does not update configuration while recreating service.#11963
#12034
Labels
kind/supportCategorizes issue or PR as a support question.lifecycle/frozenIndicates that an issue or PR should not be auto-closed due to staleness.needs-priorityneeds-triageIndicates an issue or PR lacks a `triage/foo` label and requires one.
@anvpetrov

Description

@anvpetrov
anvpetrov
opened on Sep 10, 2024

What happened:
Nginx Controller does not update configuration while recreating service.
The problem reproduces under certain conditions:

  1. Ingress object with tsl-passthrough
  2. Delete deployment and service
  3. Recreate deployment and service (as fast as possible)

What you expected to happen:
Application accesable via ingress url

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):
v1.10.1
v1.11.2
and any other

Kubernetes version (use kubectl version):
root@vm1:~# kubectl version
Client Version: v1.30.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.0
It is also Reproduced on 1.27 and other
Environment:
Dev, test, prod

OS
root@vm1:~# cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.1 LTS"

  • Kernel (e.g. uname -a):
    root@vm1:~# uname -a
    Linux vm1 5.15.0-119-generic Improve how errors connecting to api-server are handled #129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

  • How was the ingress-nginx-controller installed:
    You can reproduce problen on minikube.

  • Current State of the controller:

root@vm1:~# kubectl describe ingressclasses
Name:         nginx
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/name=ingress-nginx
Annotations:  ingressclass.kubernetes.io/is-default-class: true
Controller:   k8s.io/ingress-nginx
Events:       <none>

 root@vm1:~# kubectl -n ingress-nginx get all -A -o wide
NAMESPACE       NAME                                            READY   STATUS      RESTARTS      AGE     IP             NODE       NOMINATED NODE   READINESS GATES
ingress-nginx   pod/ingress-nginx-admission-create-slvdf        0/1     Completed   0             29m     10.244.0.5     minikube   <none>           <none>
ingress-nginx   pod/ingress-nginx-admission-patch-vv6gt         0/1     Completed   1             29m     10.244.0.4     minikube   <none>           <none>
ingress-nginx   pod/ingress-nginx-controller-5b787686df-gwzj6   1/1     Running     0             9m45s   10.244.0.7     minikube   <none>           <none>
kube-system     pod/coredns-7db6d8ff4d-d5q7q                    1/1     Running     0             30m     10.244.0.2     minikube   <none>           <none>
kube-system     pod/coredns-7db6d8ff4d-mbw2q                    1/1     Running     0             30m     10.244.0.3     minikube   <none>           <none>
kube-system     pod/etcd-minikube                               1/1     Running     0             30m     192.168.49.2   minikube   <none>           <none>
kube-system     pod/kube-apiserver-minikube                     1/1     Running     0             30m     192.168.49.2   minikube   <none>           <none>
kube-system     pod/kube-controller-manager-minikube            1/1     Running     0             31m     192.168.49.2   minikube   <none>           <none>
kube-system     pod/kube-proxy-fdgdt                            1/1     Running     0             30m     192.168.49.2   minikube   <none>           <none>
kube-system     pod/kube-scheduler-minikube                     1/1     Running     0             30m     192.168.49.2   minikube   <none>           <none>
kube-system     pod/storage-provisioner                         1/1     Running     1 (30m ago)   30m     192.168.49.2   minikube   <none>           <none>

NAMESPACE       NAME                                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE   SELECTOR
default         service/kubernetes                           ClusterIP   10.96.0.1        <none>        443/TCP                      31m   <none>
ingress-nginx   service/ingress-nginx-controller             NodePort    10.100.205.88    <none>        80:31454/TCP,443:30567/TCP   29m   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
ingress-nginx   service/ingress-nginx-controller-admission   ClusterIP   10.111.234.227   <none>        443/TCP                      29m   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
kube-system     service/kube-dns                             ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP,9153/TCP       31m   k8s-app=kube-dns

NAMESPACE     NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE   CONTAINERS   IMAGES                               SELECTOR
kube-system   daemonset.apps/kube-proxy   1         1         1       1            1           kubernetes.io/os=linux   31m   kube-proxy   registry.k8s.io/kube-proxy:v1.30.0   k8s-app=kube-proxy

NAMESPACE       NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                                                                                                                     SELECTOR
ingress-nginx   deployment.apps/ingress-nginx-controller   1/1     1            1           29m   controller   registry.k8s.io/ingress-nginx/controller:v1.10.1@sha256:e24f39d3eed6bcc239a56f20098878845f62baa34b9f2be2fd2c38ce9fb0f29e   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
kube-system     deployment.apps/coredns                    2/2     2            2           31m   coredns      registry.k8s.io/coredns/coredns:v1.11.1                                                                                    k8s-app=kube-dns

NAMESPACE       NAME                                                  DESIRED   CURRENT   READY   AGE     CONTAINERS   IMAGES                                                                                                                     SELECTOR
ingress-nginx   replicaset.apps/ingress-nginx-controller-5b787686df   1         1         1       9m46s   controller   registry.k8s.io/ingress-nginx/controller:v1.10.1@sha256:e24f39d3eed6bcc239a56f20098878845f62baa34b9f2be2fd2c38ce9fb0f29e   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=5b787686df
ingress-nginx   replicaset.apps/ingress-nginx-controller-768f948f8f   0         0         0       29m     controller   registry.k8s.io/ingress-nginx/controller:v1.10.1@sha256:e24f39d3eed6bcc239a56f20098878845f62baa34b9f2be2fd2c38ce9fb0f29e   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,pod-template-hash=768f948f8f
kube-system     replicaset.apps/coredns-7db6d8ff4d                    2         2         2       30m     coredns      registry.k8s.io/coredns/coredns:v1.11.1                                                                                    k8s-app=kube-dns,pod-template-hash=7db6d8ff4d

NAMESPACE       NAME                                       STATUS     COMPLETIONS   DURATION   AGE   CONTAINERS   IMAGES                                                                                                                              SELECTOR
ingress-nginx   job.batch/ingress-nginx-admission-create   Complete   1/1           36s        29m   create       registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.1@sha256:36d05b4077fb8e3d13663702fa337f124675ba8667cbd949c03a8e8ea6fa4366   batch.kubernetes.io/controller-uid=e80da85a-d5b7-41cb-a1d2-310539779154
ingress-nginx   job.batch/ingress-nginx-admission-patch    Complete   1/1           35s        29m   patch        registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.1@sha256:36d05b4077fb8e3d13663702fa337f124675ba8667cbd949c03a8e8ea6fa4366   batch.kubernetes.io/controller-uid=3463fa08-6ccc-42ac-9d65-f9908d84a943
root@vm1:~# kubectl -n ingress-nginx describe po ingress-nginx-controller-5b787686df-gwzj6
Name:             ingress-nginx-controller-5b787686df-gwzj6
Namespace:        ingress-nginx
Priority:         0
Service Account:  ingress-nginx
Node:             minikube/192.168.49.2
Start Time:       Tue, 10 Sep 2024 17:54:37 +0000
Labels:           app.kubernetes.io/component=controller
                  app.kubernetes.io/instance=ingress-nginx
                  app.kubernetes.io/name=ingress-nginx
                  gcp-auth-skip-secret=true
                  pod-template-hash=5b787686df
Annotations:      <none>
Status:           Running
IP:               10.244.0.7
IPs:
  IP:           10.244.0.7
Controlled By:  ReplicaSet/ingress-nginx-controller-5b787686df
Containers:
  controller:
    Container ID:  docker://154dd2e5c09ae5440a3bf4d9f5047a5e9794aaef41013d93511f614952fbea04
    Image:         registry.k8s.io/ingress-nginx/controller:v1.10.1@sha256:e24f39d3eed6bcc239a56f20098878845f62baa34b9f2be2fd2c38ce9fb0f29e
    Image ID:      docker-pullable://registry.k8s.io/ingress-nginx/controller@sha256:e24f39d3eed6bcc239a56f20098878845f62baa34b9f2be2fd2c38ce9fb0f29e
    Ports:         80/TCP, 443/TCP, 8443/TCP
    Host Ports:    80/TCP, 443/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --election-id=ingress-nginx-leader
      --controller-class=k8s.io/ingress-nginx
      --watch-ingress-without-class=true
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
      --udp-services-configmap=$(POD_NAMESPACE)/udp-services
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
      --enable-ssl-passthrough=true
    State:          Running
      Started:      Tue, 10 Sep 2024 17:54:41 +0000
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-5b787686df-gwzj6 (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-sr6jw (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       True
  ContainersReady             True
  PodScheduled                True
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  kube-api-access-sr6jw:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
                             minikube.k8s.io/primary=true
Tolerations:                 node-role.kubernetes.io/master:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age   From                      Message
  ----     ------            ----  ----                      -------
  Warning  FailedScheduling  10m   default-scheduler         0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
  Normal   Scheduled         10m   default-scheduler         Successfully assigned ingress-nginx/ingress-nginx-controller-5b787686df-gwzj6 to minikube
  Normal   Pulled            10m   kubelet                   Container image "registry.k8s.io/ingress-nginx/controller:v1.10.1@sha256:e24f39d3eed6bcc239a56f20098878845f62baa34b9f2be2fd2c38ce9fb0f29e" already present on machine
  Normal   Created           10m   kubelet                   Created container controller
  Normal   Started           10m   kubelet                   Started container controller
  Normal   RELOAD            10m   nginx-ingress-controller  NGINX reload triggered due to a change in configuration
root@vm1:~# kubectl -n ingress-nginx describe svc ingress-nginx-controller
Name:                     ingress-nginx-controller
Namespace:                ingress-nginx
Labels:                   app.kubernetes.io/component=controller
                          app.kubernetes.io/instance=ingress-nginx
                          app.kubernetes.io/name=ingress-nginx
Annotations:              <none>
Selector:                 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type:                     NodePort
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.100.205.88
IPs:                      10.100.205.88
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  31454/TCP
Endpoints:                10.244.0.7:80
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  30567/TCP
Endpoints:                10.244.0.7:443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

How to reproduce this issue**:

#Install minikube
minikube start --force
#Install Ingress
minikube addons enable ingress
#Edit ingress deployment for enable passthrough
kubectl edit deployments.apps -n ingress-nginx ingress-nginx-controller
Add arg
--enable-ssl-passthrough=true

Install an application

kubectl create ns test
kubectl create deployment web --image=gcr.io/google-samples/hello-app:1.0 -n test
kubectl expose deployment web --port=8080 -n test

Create an ingress (please add any additional annotation required)

echo "
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: 'HTTPS'
    nginx.ingress.kubernetes.io/enable-access-log: 'true'
    nginx.ingress.kubernetes.io/ssl-passthrough: 'true'
  name: example-ingress-https
  namespace: test
spec:
  ingressClassName: nginx
  rules:
  - host: hello-world-https.example
    http:
      paths:
      - backend:
          service:
            name: web
            port:
              number: 8080
        path: /
        pathType: Prefix
" | kubectl apply -f -

make a request

Check that its working
curl -k --resolve "hello-world-https.example:443:$( minikube ip )" -i https://hello-world-https.example
curl: (35) error:0A00010B:SSL routines::wrong version number

SSL error because our application is running over HTTP, this is normal.

We will prepare the manifests:

cat <<EOF > manifests.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: web
  name: web
  namespace: test
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: web
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: web
    spec:
      containers:
      - image: gcr.io/google-samples/hello-app:1.0
        imagePullPolicy: IfNotPresent
        name: hello-app
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: web
  name: web
  namespace: test
spec:
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: web
  sessionAffinity: None
  type: ClusterIP
EOF

cat <<EOF > run.sh
kubectl delete deployment web -n test
kubectl delete service web -n test
kubectl create -f manifests.yaml
EOF

Now, we cat start test.

  1. Check service
    kubectl get svc -n test
    NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
    web ClusterIP 10.101.12.157 8080/TCP 3m15s

IP 10.101.12.157

Run our bash script:
bash run.sh
deployment.apps "web" deleted
service "web" deleted
deployment.apps/web created
service/web created

Check again:
curl -k --resolve "hello-world-https.example:443:$( minikube ip )" -i https://hello-world-https.example
curl: (35) error:0A000126:SSL routines::unexpected eof while reading

If we run tcpdump, we can see, that packets send to old service IP
tcpdump -nnvvS -i any host 10.101.12.157
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
18:56:27.059554 vethda6e844 P IP (tos 0x0, ttl 63, id 50339, offset 0, flags [DF], proto TCP (6), length 60)
192.168.49.2.57792 > 10.101.12.157.8080: Flags [S], cksum 0x08db (incorrect -> 0x20bc), seq 3739768812, win 64240, options [mss 1460,sackOK,TS val 4031445813 ecr 0,nop,wscale 7], length 0
18:56:27.059554 br-8f45197f04bc In IP (tos 0x0, ttl 63, id 50339, offset 0, flags [DF], proto TCP (6), length 60)
192.168.49.2.57792 > 10.101.12.157.8080: Flags [S], cksum 0x08db (incorrect -> 0x20bc), seq 3739768812, win 64240, options [mss 1460,sackOK,TS val 4031445813 ecr 0,nop,wscale 7], length 0
18:56:27.059571 enp0s3 Out IP (tos 0x0, ttl 62, id 50339, offset 0, flags [DF], proto TCP (6), length 60)
10.0.2.15.57792 > 10.101.12.157.8080: Flags [S], cksum 0x233f (incorrect -> 0x0658), seq 3739768812, win 64240, options [mss 1460,sackOK,TS val 4031445813 ecr 0,nop,wscale 7], length 0
18:56:28.066013 vethda6e844 P IP (tos 0x0, ttl 63, id 50340, offset 0, flags [DF], proto TCP (6), length 60)
192.168.49.2.57792 > 10.101.12.157.8080: Flags [S], cksum 0x08db (incorrect -> 0x1ccd), seq 3739768812, win 64240, options [mss 1460,sackOK,TS val 4031446820 ecr 0,nop,wscale 7], length 0
18:56:28.066013 br-8f45197f04bc In IP (tos 0x0, ttl 63, id 50340, offset 0, flags [DF], proto TCP (6), length 60)
192.168.49.2.57792 > 10.101.12.157.8080: Flags [S], cksum 0x08db (incorrect -> 0x1ccd), seq 3739768812, win 64240, options [mss 1460,sackOK,TS val 4031446820 ecr 0,nop,wscale 7], length 0

If you change Ingress after that script, then It will work.

Metadata

Assignees

No one assigned

    Labels

    kind/supportCategorizes issue or PR as a support question.lifecycle/frozenIndicates that an issue or PR should not be auto-closed due to staleness.needs-priorityneeds-triageIndicates an issue or PR lacks a `triage/foo` label and requires one.

    Type

    No type

    Projects

    [SIG Network] Ingress NGINX

    • Status

      No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions