What happened:

SSL certificate on backend was not verified despite providing annotations.

I0801 10:53:39.076948       7 annotations.go:184] "Parsing Ingress annotation" name="ProxySSL" ingress="OMITTED/OMITTED" value={"secret":"","caFilename":"","caSha":"","crlFileName":"","crlSha":"","pemFilename":"","ciphers":"","protocols":"","proxySSLName":"","verify":"","verifyDepth":0,"proxySSLServerName":""}

What you expected to happen:

I expected options verifyDepth and verify to be configured.

I discovered the cause in proxyssl/main.go#L119.

In the absence of client certificate data (secret, certificate), no other options are considered. The missing annotation error is reported to the calling function but that function (perhaps correctly) does not consider this an error.

The docs describe the use of these annotations for client authentication, but I do not need client authentication. My primary goal is to ensure that I have step-wise encryption and no unexpected man-in-the-middle attack. So I need each step in my flow to: 1) use TLS; 2) verify target server; 3) verify that a trusted root certificate was used.

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

-------------------------------------------------------------------------------

NGINX Ingress controller

  Release:       v1.8.0

  Build:         35f5082ee7f211555aaff431d7c4423c17f8ce9e

  Repository:    https://github.com/kubernetes/ingress-nginx

  nginx version: nginx/1.21.6



-------------------------------------------------------------------------------

Kubernetes version (use kubectl version):

WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.

Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.2", GitCommit:"7f6f68fdabc4df88cfea2dcf9a19b2b830f1e647", GitTreeState:"clean", BuildDate:"2023-05-17T14:20:07Z", GoVersion:"go1.20.4", Compiler:"gc", Platform:"linux/amd64"}

Kustomize Version: v5.0.1

Server Version: version.Info{Major:"1", Minor:"27+", GitVersion:"v1.27.3-eks-a5565ad", GitCommit:"78c8293d1c65e8a153bf3c03802ab9358c0e1a14", GitTreeState:"clean", BuildDate:"2023-06-16T17:32:40Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: AWS, EKS

• OS (e.g. from /etc/os-release): Amazon Linux 2

• Kernel (e.g. uname -a): 5.10.179-166.674.amzn2.x86_64 #1 SMP Mon May 8 16:54:25 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

• Install tools:

  • Please mention how/where was the cluster created like kubeadm/kops/minikube/kind etc.
  • Terraform scripts, custom node scripts based on AWS EKS node template AMI

• Basic cluster related info:

  • kubectl version
  • see above
  • kubectl get nodes -o wide
  • not disclosed

• How was the ingress-nginx-controller installed:

  • If helm was used then please show output of helm ls -A | grep -i ingress
  • If helm was used then please show output of helm -n <ingresscontrollernamepspace> get values <helmreleasename>
  • If helm was not used, then copy/paste the complete precise command used to install the controller, along with the flags and options used
  • if you have more than one instance of the ingress-nginx-controller installed in the same cluster, please provide details for all the instances
  • manifests adapted from GitHub; do not use Helm

• Current State of the controller:

  • This is not sanitized. I don't think it is necessary for this issue. The code for the behaviour is easy to interpret.

• Current state of ingress object, if applicable:

  • Annotations added to ingress:

    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/proxy-ssl-verify: "on"
    nginx.ingress.kubernetes.io/proxy-ssl-verify-depth: "10"
    nginx.ingress.kubernetes.io/proxy-ssl-server-name: "on"

• Others:
  • Any other related information like ;
    • copy/paste of the snippet (if applicable)
    • kubectl describe ... of any custom configmap(s) created and in use
    • Any other related information that may help

How to reproduce this issue:

• Edit the controller: add the log level --v=5 to include debug level logging
• Create a workload with a pod and service exposing HTTPS
• Create an ingress for HTTPS pointing to the service. Ensure the service has the following annotations:

    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/proxy-ssl-verify: "on"
    nginx.ingress.kubernetes.io/proxy-ssl-verify-depth: "10"
    nginx.ingress.kubernetes.io/proxy-ssl-server-name: "on"

• Review the controller logs to see that the proxy-ssl-* annotations have been ignored

Anything else we need to know:

Would be good to make it clear in the annotation documentation that these are a group that have to be configured as a single set.