Getting ingress controller fake certificate in ssl-passthrough mode

[feiluo-db]

I deployed the ingress controller with --enable-ssl-passthrough flag on. Verified in the nginx.conf file that it is indeed turned on.
Ingress controller is started with --ingress-class=my-test-nginx to match the ingress class annotation on Ingress resource.
On my ingress resource annotation, I added

"nginx.ingress.kubernetes.io/ssl-passthrough": "true"

The full configuration looks like the following

{
    apiVersion: "networking.k8s.io/v1",
    kind: "Ingress",
    metadata: {
      name: "my-test-ingress",
      namespace: "my-test-ns",
      annotations: {
        "kubernetes.io/ingress.class": "my-test-nginx",
        "nginx.ingress.kubernetes.io/ssl-passthrough": "true",
        "nginx.ingress.kubernetes.io/ssl-redirect": "true",
      },
    },
    spec: {
      rules: [
        {
          host: "my-test.dev.example.com",
          http: {
            paths: [
              {
                path: "/",
                pathType: "Prefix",
                backend: {
                  service: {
                    name: "my-test-svc",
                    port: {
                      number: 8443,
                    },
                  },
                },
              },
            ],
          },
        },
      ],
    },
  },
{
    appName:: "my-test-svc",
    apiVersion: "v1",
    kind: "Service",

    metadata: {
      name: "my-test-svc",
      namespace: "my-test-ns",
    },
    spec: {
      ports: [
        {
          name: "doesnt matter",
          port: 8443,
          targetPort: 8443,
          protocol: "TCP",
        },
      ],

      selector: { app: "my-test-app" },
      type: "ClusterIP",
    },
  },

My ingress controller is deployed on AWS as a AWS ELB. No TLS cert is configured on the listener of ELB as it shouldn't terminate TLS.
Any advice on how to further debug this would be very much appreciated!

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[longwuyuan]

show the curl request and the response with flags -iv.

if you answer the questions asked in a new issue template, then the readers can have data to analyze and base comments on.

[feiluo-db]

I noticed a mis-configuration. Sorry for the noise!

[feiluo-db]

Actually it still doesn't work.

curl request and response with flags -iv

curl -iv https://runbot-fei-luo-new-8-ci-shard.dev.databricks.com
*   Trying 44.227.188.95:443...
* TCP_NODELAY set
* Connected to runbot-fei-luo-new-8-ci-shard.dev.databricks.com (44.227.188.95) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

[longwuyuan]

@feiluo-db you will still get only guess based comments.

You showed the curl command and output.

But unless yo ushow the kubectl describe command output of all related resources like ;

• kubectl -n ingress-nginx get all
• kubectl -n ingress-nginx describe svc ingress-nginx-controller
• kubectl -n my-test-ns get all
• kubectl -n my-test-ns describe svc my-test-svc
• kubectl -n my-test-ns describe my-test-ingress

In the post above, you are sending request to a host different from the one sen in the ingress yaml so the data you sent is useless for analysis.

[feiluo-db]

@longwuyuan Yeah the my-test names are fake names I made up for simplicity.

Here are the actual outputs:

k describe svc cci-ingress-controller-service

Name:                        cci-ingress-controller-service
Namespace:                   jenkins-build
Labels:                      app.kubernetes.io/component=controller
                             app.kubernetes.io/instance=ingress-nginx
                             app.kubernetes.io/name=ingress-nginx
                             app.kubernetes.io/part-of=ingress-nginx
                             app.kubernetes.io/version=0.41.0
Annotations:            
                           
                             service.beta.kubernetes.io/aws-load-balancer-type: elb
Selector:                    app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,app=cci-ingress-controller
Type:                        LoadBalancer
IP Family Policy:            SingleStack
IP Families:                 IPv4
IP:                          10.3.203.29
IPs:                         10.3.203.29
LoadBalancer Ingress:        a70fe2029834541878f29b5f25767d0f-557761935.us-west-2.elb.amazonaws.com
Port:                        metrics  10254/TCP
TargetPort:                  10254/TCP
NodePort:                    metrics  31293/TCP
Endpoints:                   10.6.23.194:10254
Port:                        http  80/TCP
TargetPort:                  80/TCP
NodePort:                    http  32145/TCP
Endpoints:                   10.6.23.194:80
Port:                        https  443/TCP
TargetPort:                  https/TCP
NodePort:                    https  32003/TCP
Endpoints:                   10.6.23.194:443
Session Affinity:            None
External Traffic Policy:     Local
HealthCheck NodePort:        31511

k describe svc runbot-fei-luo-new-8-ci-shard-internal-ext

Name:              runbot-fei-luo-new-8-ci-shard-internal-ext
Namespace:         jenkins-build
Labels:            <none>
Annotations:       databricks/last_modified_by: fei.luo
Selector:          app=runbot-fei-luo-new-8-ci-shard
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.3.158.7
IPs:               10.3.158.7
Port:              runbot-port-ingress  8443/TCP
TargetPort:        8443/TCP
Endpoints:         10.6.16.174:8443
Session Affinity:  None
Events:            <none>

k describe ing runbot-fei-luo-new-8-ci-shard-ingress

Name:             runbot-fei-luo-new-8-ci-shard-ingress
Labels:           <none>
Namespace:        jenkins-build
Address:          
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host                                              Path  Backends
  ----                                              ----  --------
  runbot-fei-luo-new-8-ci-shard.dev.databricks.com  
                                                    /   runbot-fei-luo-new-8-ci-shard-internal-ext:8443 (10.6.16.174:8443)
Annotations:                                        kubernetes.io/ingress.class: ci-shard-nginx
                                                    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
                                                    nginx.ingress.kubernetes.io/ssl-passthrough: true
Events:
  Type    Reason  Age    From                      Message
  ----    ------  ----   ----                      -------
  Normal  Sync    4m30s  nginx-ingress-controller  Scheduled for sync
  Normal  Sync    4m28s  nginx-ingress-controller  Scheduled for sync

[longwuyuan]

There are 2 problems. First change your ingress annotations as per this example https://argo-cd.readthedocs.io/en/stable/operator-manual/ingress/#kubernetesingress-nginx. Second prove that your jenkins server is using a certificate with well known CA. But also important to note is that you are using a unsupported version of the controller. It will work but we don't support that old version so upgrade to latest and test.

…

On Wed, 26 Feb, 2025, 10:51 Fei Luo, ***@***.***> wrote: @longwuyuan <https://github.com/longwuyuan> Yeah the my-test names are fake names I made up for simplicity. Here are the actual outputs: *k describe svc cci-ingress-controller-service* Name: cci-ingress-controller-service Namespace: jenkins-build Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=0.41.0 Annotations: service.beta.kubernetes.io/aws-load-balancer-type: elb Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,app=cci-ingress-controller Type: LoadBalancer IP Family Policy: SingleStack IP Families: IPv4 IP: 10.3.203.29 IPs: 10.3.203.29 LoadBalancer Ingress: a70fe2029834541878f29b5f25767d0f-557761935.us-west-2.elb.amazonaws.com Port: metrics 10254/TCP TargetPort: 10254/TCP NodePort: metrics 31293/TCP Endpoints: 10.6.23.194:10254 Port: http 80/TCP TargetPort: 80/TCP NodePort: http 32145/TCP Endpoints: 10.6.23.194:80 Port: https 443/TCP TargetPort: https/TCP NodePort: https 32003/TCP Endpoints: 10.6.23.194:443 Session Affinity: None External Traffic Policy: Local HealthCheck NodePort: 31511 *k describe svc runbot-fei-luo-new-8-ci-shard-internal-ext* Name: runbot-fei-luo-new-8-ci-shard-internal-ext Namespace: jenkins-build Labels: <none> Annotations: databricks/last_modified_by: fei.luo Selector: app=runbot-fei-luo-new-8-ci-shard Type: ClusterIP IP Family Policy: SingleStack IP Families: IPv4 IP: 10.3.158.7 IPs: 10.3.158.7 Port: runbot-port-ingress 8443/TCP TargetPort: 8443/TCP Endpoints: 10.6.16.174:8443 Session Affinity: None Events: <none> *k describe ing runbot-fei-luo-new-8-ci-shard-ingress* Name: runbot-fei-luo-new-8-ci-shard-ingress Labels: <none> Namespace: jenkins-build Address: Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>) Rules: Host Path Backends ---- ---- -------- runbot-fei-luo-new-8-ci-shard.dev.databricks.com / runbot-fei-luo-new-8-ci-shard-internal-ext:8443 (10.6.16.174:8443) Annotations: kubernetes.io/ingress.class: ci-shard-nginx nginx.ingress.kubernetes.io/backend-protocol: HTTPS nginx.ingress.kubernetes.io/ssl-passthrough: true Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Sync 4m30s nginx-ingress-controller Scheduled for sync Normal Sync 4m28s nginx-ingress-controller Scheduled for sync — Reply to this email directly, view it on GitHub <#12897 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGZVWRX6D5KFAUYHSPJRD32RVFL3AVCNFSM6AAAAABXZEZENOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOBTHEZDINZRGI> . You are receiving this because you were mentioned.Message ID: ***@***.***> [image: feiluo-db]*feiluo-db* left a comment (kubernetes/ingress-nginx#12897) <#12897 (comment)> @longwuyuan <https://github.com/longwuyuan> Yeah the my-test names are fake names I made up for simplicity. Here are the actual outputs: *k describe svc cci-ingress-controller-service* Name: cci-ingress-controller-service Namespace: jenkins-build Labels: app.kubernetes.io/component=controller app.kubernetes.io/instance=ingress-nginx app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx app.kubernetes.io/version=0.41.0 Annotations: service.beta.kubernetes.io/aws-load-balancer-type: elb Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx,app=cci-ingress-controller Type: LoadBalancer IP Family Policy: SingleStack IP Families: IPv4 IP: 10.3.203.29 IPs: 10.3.203.29 LoadBalancer Ingress: a70fe2029834541878f29b5f25767d0f-557761935.us-west-2.elb.amazonaws.com Port: metrics 10254/TCP TargetPort: 10254/TCP NodePort: metrics 31293/TCP Endpoints: 10.6.23.194:10254 Port: http 80/TCP TargetPort: 80/TCP NodePort: http 32145/TCP Endpoints: 10.6.23.194:80 Port: https 443/TCP TargetPort: https/TCP NodePort: https 32003/TCP Endpoints: 10.6.23.194:443 Session Affinity: None External Traffic Policy: Local HealthCheck NodePort: 31511 *k describe svc runbot-fei-luo-new-8-ci-shard-internal-ext* Name: runbot-fei-luo-new-8-ci-shard-internal-ext Namespace: jenkins-build Labels: <none> Annotations: databricks/last_modified_by: fei.luo Selector: app=runbot-fei-luo-new-8-ci-shard Type: ClusterIP IP Family Policy: SingleStack IP Families: IPv4 IP: 10.3.158.7 IPs: 10.3.158.7 Port: runbot-port-ingress 8443/TCP TargetPort: 8443/TCP Endpoints: 10.6.16.174:8443 Session Affinity: None Events: <none> *k describe ing runbot-fei-luo-new-8-ci-shard-ingress* Name: runbot-fei-luo-new-8-ci-shard-ingress Labels: <none> Namespace: jenkins-build Address: Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>) Rules: Host Path Backends ---- ---- -------- runbot-fei-luo-new-8-ci-shard.dev.databricks.com / runbot-fei-luo-new-8-ci-shard-internal-ext:8443 (10.6.16.174:8443) Annotations: kubernetes.io/ingress.class: ci-shard-nginx nginx.ingress.kubernetes.io/backend-protocol: HTTPS nginx.ingress.kubernetes.io/ssl-passthrough: true Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Sync 4m30s nginx-ingress-controller Scheduled for sync Normal Sync 4m28s nginx-ingress-controller Scheduled for sync — Reply to this email directly, view it on GitHub <#12897 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGZVWRX6D5KFAUYHSPJRD32RVFL3AVCNFSM6AAAAABXZEZENOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOBTHEZDINZRGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[feiluo-db]

Thanks. Yes I did follow https://argo-cd.readthedocs.io/en/stable/operator-manual/ingress/#kubernetesingress-nginx to add the annotations to the ingress resource. It didn't work if I remember correctly. I can try that again but likely it won't solve the issue.

Secondly, I curl'ed the service cluster IP from within the same cluster to make sure the server certificate is valid. I did get the right server certificate which has DigiCert as root CA in the chain.

I think the ingress controller image we're using is v-1.3.1. I can try to upgrade to a newer version, but again I don't think it is the root cause as I think ssl-passthrough should be well supported even before that version.