namespaced ingress doesn't work as expected

[yong-jie-gong]

What happened:
From kubernetes 1.18, kubernetes deprecate ingress annotation "kubernetes.io/ingress.class", instead, it is replaced with ingress.Spec.IngressClass. for cluster Ingress, it is ok. but for namespaced ingress. cluster don't want to grant any cluster resource permission to ingress-controller. it means nginx-ingrss-controller have no permissions to access the IngressClass object. in current nginx-ingress-controller, it mandate the IngressClass existence referred as ingress.Spec.IngressClassName.

As a result, for Namespaced ingress scenario, ingress annnotation "kubernetes.io/ingress.class" is the only choice. it works at this time, but from kubernets 1.28, kubernetes server keep printing warning if ingress has annotation "kubernetes.io/ingress.class". it is not ideal.

What you expected to happen:

so it is better support namespaced ingressClass without accessing the IngresClass object and using the annotation.
suggestions:

1. IngressController needn't cluster level permission to access the IngressClass for namespaced Ingress
2. consumer drop annotation "kubernetes.io/ingress.class" from ingress
3. Consumer set the ingressClassName by ingress.Spec.IngressClassName
4. IngressController accept the incoming ingress object when
   1. IngressController has permission to IngressClass, keep the current implementation.
   2. IngressController dont' have permission to access the IngressClass but ingress.Spec.IngressClassName is equals to the ingress class name specified by CLI parameter "--ingress-class"

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

Kubernetes version (use kubectl version): v1.29.2

Environment:

• Cloud provider or hardware configuration:

• OS (e.g. from /etc/os-release):

• Kernel (e.g. uname -a):

• Install tools:

  • Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc.

• Basic cluster related info:

  • kubectl version
  • kubectl get nodes -o wide

• How was the ingress-nginx-controller installed:

  • If helm was used then please show output of helm ls -A | grep -i ingress
  • If helm was used then please show output of helm -n <ingresscontrollernamespace> get values <helmreleasename>
  • If helm was not used, then copy/paste the complete precise command used to install the controller, along with the flags and options used
  • if you have more than one instance of the ingress-nginx-controller installed in the same cluster, please provide details for all the instances
    /nginx-ingress-controller --kubeconfig=/root/.kube/config
    --default-ssl-certificate=core/demo1-nginx-secret
    --v=0
    --configmap=core/demo1-ingress-controller-conf
    --watch-namespace=core
    --annotations-prefix=ingress.kubernetes.io
    --enable-ssl-chain-completion=false
    --http-port=8080
    --https-port=8443
    --enable-annotation-validation=true
    --update-status=false
    --ingress-class=demo1-nginx
    --metrics-per-host=false
    --enable-metrics=false "

• Current State of the controller:

  • kubectl describe ingressclasses
  • kubectl -n <ingresscontrollernamespace> get all -A -o wide
  • kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>
  • kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>

• Current state of ingress object, if applicable:

  • kubectl -n <appnamespace> get all,ing -o wide
  • kubectl -n <appnamespace> describe ing <ingressname>
  • If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag

• Others:

  • Any other related information like ;
    • copy/paste of the snippet (if applicable)
    • kubectl describe ... of any custom configmap(s) created and in use
    • Any other related information that may help

How to reproduce this issue:

Anything else we need to know:

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[yong-jie-gong]

raise PR #11223

[longwuyuan]

• I can't find documentation that describes installing namespace scoped instance

• Before Kubernetes v1.24, there used to be attempts and use-cases of namespace scoped installation

• Even though you mentioned that some cluster-admins do not allow users to access cluster-wide resources, the upstream K8S Ingress API based design of this controller involves to access to cluster-wide resources. So I don't think its a improvement to change the the behavior of the controller to restrict access to a namespace

• There are far too important and huge number of users who use the annotation for the ingressClassName, particularly cert-manager. So this proiect has to continue support for the annotation

• I don't see the practical benefit of the change you suggest here because I don't see how a ingress-controller is appropriate to be running in a cluster without the cluster-admin's approval/consent & co-operation. The PR you submitted does not have a description of the solution. At least describe the entire solution in small details so that a valid case is presented to the reader

• I see a hard change just to 2 go files, without any consideration to how it will impact a user's experience and tests to show the working of the changes.

• Does your changes impact the rest of the controller's features like --default-ssl-certificate etc,

[longwuyuan]

/remove-kind bug

[yong-jie-gong]

@longwuyuan thanks for your quick response. as requested in PR #11223

Assume that the sample application to be deployed is --image nginx:alpine
kubectl create deployment test0 --image nginx:alpine --port 80
Assume that the service for this is kubectl expose deployment test0 --port 80
Now write a ingress resource yaml file for it and keep it ready for use after the clusrter is ready
Create a minikube cluster
Fork the project on github
create a branch and clone
Make your changes to the code
Run make dev-env
Now there will be a cluster ready with your changes to the controller code
Deploy your app and service and ingress
Copy/paste all the test and logs and state related info as outputs of commands here on in the issue
Then I will have more practical ways to copy your fork's branch and do the same and test your changed controller locally
I can then put the default-ssl-certificate in a different namespace and see how I can configure ingress with TLS but without a cert
I can then see first hand what you mean by not-using-cluster-ingress-class

please check information from my env as below

1. Setup dev test with "make dev-env".
   Kubernetes cluster ready and ingress-nginx listening in localhost using ports 80 and 443
   To delete the dev cluster execute: 'kind delete cluster --name ingress-nginx-dev'

# kind get clusters
ingress-nginx-dev

# kubectl get po
NAME                                        READY   STATUS      RESTARTS   AGE
ingress-nginx-admission-create-2vf6g        0/1     Completed   0          16m
ingress-nginx-admission-patch-cxtx8         0/1     Completed   2          16m
ingress-nginx-controller-659c6c4948-pr8jm   1/1     Running     0          16m
test0-574c47cb97-fzhjf                      1/1     Running     0          8m11s

2. by default, one "ingress-nginx-controller" is deployed in my env whose service account is bound to cluster role "" below

# kubectl get clusterrolebinding |grep ingres
ingress-nginx                                          ClusterRole/ingress-nginx                                                          18m
ingress-nginx-admission                                ClusterRole/ingress-nginx-admission                                                18m

# kubectl get clusterrolebinding ingress-nginx  -ojsonpath='{.roleRef} {"\n"} {.subjects}'
 {"apiGroup":"rbac.authorization.k8s.io","kind":"ClusterRole","name":"ingress-nginx"}
 [{"kind":"ServiceAccount","name":"ingress-nginx","namespace":"ingress-nginx"}]

3) for namespaced deployment, nginx-ingress-controller is not supposed to have cluster level permission. so remove cluster rolebinding "ingress-nginx"

# kubectl delete clusterrolebinding  ingress-nginx
clusterrolebinding.rbac.authorization.k8s.io "ingress-nginx" deleted

4. default nginx-ingress-controller pod watches the whole cluster, so update nginx-ingress-controller deployment add the CLI parameter "- --watch-namespace=$(POD_NAMESPACE)"

5. deployment my service/app/ingress as requested from your
   more test0.svc.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    app: test0
  name: test0
  namespace: ingress-nginx
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: test0
  type: ClusterIP

more test0.deploy.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: test0
  name: test0
  namespace: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test0
  template:
    metadata:
      labels:
        app: test0
    spec:
      containers:
      - image: nginx:alpine
        imagePullPolicy: IfNotPresent
        name: nginx
        ports:
        - containerPort: 80
          protocol: TCP
      restartPolicy: Always

more test0.ing.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-class-name-no-perm
  namespace: ingress-nginx
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - backend:
          service:
            name: test0
            port:
              number: 80
        path: /demo/http1
        pathType: Prefix
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-invalid-ingress-class-name-no-perm
  namespace: ingress-nginx
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx-not-match
  rules:
  - http:
      paths:
      - backend:
          service:
            name: test0
            port:
              number: 80
        path: /demo/http2
        pathType: Prefix
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx-annotation
    nginx.ingress.kubernetes.io/rewrite-target: /
  name: ingress-from-annotation
  namespace: ingress-nginx
spec:
  rules:
  - http:
      paths:
      - backend:
          service:
            name: test0
            port:
              number: 80
        path: /demo/http3
        pathType: Prefix

6. check nginx-ingress-controller pod output which is not trying to watch cluter level resource IngressClass "No permissions to list and get Ingress Classes:"

# kubectl logs ingress-nginx-controller-7fd476c957-6t5cp

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       1.0.0-dev
  Build:         git-7c2b047c5
  Repository:    git@github.com:yong-jie-gong/ingress-nginx.git
  nginx version: nginx/1.25.3

-------------------------------------------------------------------------------

W0415 08:06:53.880250      12 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0415 08:06:53.880720      12 main.go:205] "Creating API client" host="https://10.96.0.1:443"
I0415 08:06:53.934795      12 main.go:248] "Running in Kubernetes cluster" major="1" minor="26" git="v1.26.3" state="clean" commit="9e644106593f3f4aa98f8a84b23db5fa378900bd" platform="linux/amd64"
I0415 08:06:55.130756      12 main.go:101] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
W0415 08:06:55.154262      12 main.go:111] No permissions to list and get Ingress Classes: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:ingress-nginx:ingress-nginx" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope, IngressClass feature will be disabled

7. check generated nginx.conf file

# kubectl get po|grep ingress-nginx-controller-7fd476c957-6t5cp
ingress-nginx-controller-7fd476c957-6t5cp   1/1     Running     0          14m

# kubectl exec -it ingress-nginx-controller-7fd476c957-6t5cp -- bash -c 'more /etc/nginx/nginx.conf|grep /demo'
                location ~* "^/demo/http1" {
                        set $location_path  "/demo/http1";
                        rewrite "(?i)/demo/http1" / break;

8) Check deployed demo application.  "/demo/http1" is picked up when "nginx-ingress-controller" pod don't have permission to access the cluster level resource "IngressClass"
# kubectl  get svc -n$ns|grep ingress
ingress-nginx-controller             NodePort    10.96.31.140   <none>        80:32691/TCP,443:31098/TCP   67m

# ip=10.96.31.140  
root@ingress-nginx-dev-control-plane:/# curl -s -o /dev/null --head --write-out '%{http_code}' --noproxy $ip http://$ip:80/demo/http1 ; echo $http_code
200
root@ingress-nginx-dev-control-plane:/# curl -s -o /dev/null --head --write-out '%{http_code}' --noproxy $ip http://$ip:80/demo/http2 ; echo $http_code
404
root@ingress-nginx-dev-control-plane:/# curl -s -o /dev/null --head --write-out '%{http_code}' --noproxy $ip http://$ip:80/demo/http3 ; echo $http_code
404

root@ingress-nginx-dev-control-plane:/# curl -s -o /dev/null --head --write-out '%{http_code}' -k --noproxy $ip https://$ip:443/demo/http1 ; echo $http_code
200
root@ingress-nginx-dev-control-plane:/# curl -s -o /dev/null --head --write-out '%{http_code}' -k --noproxy $ip https://$ip:443/demo/http2 ; echo $http_code
404
root@ingress-nginx-dev-control-plane:/# curl -s -o /dev/null --head --write-out '%{http_code}' -k --noproxy $ip https://$ip:443/demo/http3 ; echo $http_code
404

[longwuyuan]

@yong-jie-gong I request some detailed information which helps reduce the work to be done by others. Is it possible for you ti kindly edit the above message and post information as per hints below ;

• When you run make make dev-env, please do it from a shell, where you have your fork+clone+branch (in which you made your changes to the controller code)

• First show git diff so that all the changes you made are visible

• Show output of following commands instead of yaml files

  • helm ls -A
  • kubectl - ingress-nginx get all
  • kubectl describe clusterrole ingress-nginx
  • kubectl describe clusterrolebindings.rbac.authorization.k8s.io ingress-nginx
  • kubectl describe sa
  • kubectl get all,ing
  • kubectl describe ing
  • kubectl get events
  • curl test0.local -v

And other such information. This is to see the live state of the resources like clusterrole and others from your changes as well the curl command and the other commands that explains how ingress is working after your changes

[longwuyuan]

/kind feature