Skip to content

Incorrect HTTPS redirection using "from-to-www-redirect" annotation and TLS Terminated Loadbalancer #12874

Open
@mreho

Description

@mreho

What happened:

When using the annotation nginx.ingress.kubernetes.io/from-to-www-redirect: 'true' on any of my Nginx ingresses, requests to https://www.example.com is incorrectly redirected to https://example.com:80, causing an access error (trying to access using TLS on plaintext HTTP port).

Behind my Kubernetes cluster, there is a Loadbalancer with two listeners.
One of them, listening on port 443, has TLS Terminating / TLS Offloading feature (it has direct access to the SSL certificates).
The Loadbalancer is setup on Openstack Octavia and is using HAProxy.

Recap :

Public trafic comes on port 443 of the LB.
Request is uncrypted then forwarded to the :80 pool.
Standard clear requests on port 80 are also forwarded to the :80 pool.

On every request, the following HTTP headers are inserted by the LB listeners :

  • X-Forwarded-For
  • X-Forwarded-Port
  • X-Forwarded-Proto

(HTTP plaintext request)
Client -----> Listener :80 ------------> Pool :80 ------ Proxy Protocol V2 -----> K8S Worker Nodes :80
                                 /
                           TLS Termination
(HTTPS encrypted request)     /
Client -----> Listener :443  /

This issue persists even with the following configuration :

controller:
  config:
    enable-real-ip: "true"
    proxy-real-ip-cidr: 10.xxx.xxx.0/24,2001:xxxx:xxxx:106::/64
    use-forwarded-headers: "true"
    use-proxy-protocol: "true"
  service:
    targetPorts:
      http: http
      https: http

Source : https://github.com/kubernetes/ingress-nginx/tree/main/charts/ingress-nginx#aws-l7-elb-with-ssl-termination

What you expected to happen:

Nginx Ingress Controller should take care of X-Forwarded-Port and X-Forwarded-Proto, to know the request was originally coming from port 443 and was HTTPS.

It should instead redirect https://www.example.com to https://example.com:443 (or simply https://example.com).

NGINX Ingress controller version :

www-data@rke2-ingress-nginx-controller-s5b5j:/etc/nginx> /nginx-ingress-controller --version
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.12.0-hardened1
  Build:         git-5bfb8f4b7
  Repository:    https://github.com/rancher/ingress-nginx
  nginx version: nginx/1.25.5

-------------------------------------------------------------------------------

Kubernetes version : v1.31.5+rke2r1

Environment:

  • Cloud provider or hardware configuration : OpenStack (hosted by Infomaniak.com)

  • OS : Ubuntu 24.04.2 LTS

  • Kernel : 6.8.0-51-generic

  • Install tools:

    • Infrastructure was setup using OpenTofu + CloudInit for nodes
    • Cluster was setup using Ansible (thanks to Labyrinth Labs)
  • Basic cluster related info:

    • 3 Master nodes (with taint NoExecute)
    • 6 Worker nodes
    • Cilium CNI (eBPF routing, Geneve tunneling, kube-proxy replacement enabled, DSR enabled, XDP enabled)
    • IPv4 / IPv6 Dualstack (fully working)
  • How was the ingress-nginx-controller installed:

$ helm ls -A | grep -i ingress
rke2-ingress-nginx          	kube-system	101     	2025-02-20 03:52:46.70060196 +0100 CET 	deployed	rke2-ingress-nginx-4.12.003         	1.12.0
  • Current State of the controller:
$ kubectl describe ingressclasses
Name:         nginx
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=rke2-ingress-nginx
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=rke2-ingress-nginx
              app.kubernetes.io/part-of=rke2-ingress-nginx
              app.kubernetes.io/version=1.12.0
              helm.sh/chart=rke2-ingress-nginx-4.12.003
Annotations:  meta.helm.sh/release-name: rke2-ingress-nginx
              meta.helm.sh/release-namespace: kube-system
Controller:   k8s.io/ingress-nginx
Events:       <none>
$ kubectl -n kube-system get all -o wide | grep ingress
pod/helm-install-rke2-ingress-nginx-k7p56                   0/1     Completed   0               4d4h   <none>         inst-k8s-worker-3   <none>           <none>
pod/rke2-ingress-nginx-controller-blmrw                     1/1     Running     0               4d4h   10.42.xxx.xxx  inst-k8s-worker-2   <none>           <none>
pod/rke2-ingress-nginx-controller-j67b9                     1/1     Running     0               4d4h   10.42.xxx.xxx  inst-k8s-worker-1   <none>           <none>
pod/rke2-ingress-nginx-controller-l22rl                     1/1     Running     0               4d4h   10.42.xxx.xxx  inst-k8s-worker-4   <none>           <none>
pod/rke2-ingress-nginx-controller-s5b5j                     1/1     Running     0               4d4h   10.42.xxx.xxx  inst-k8s-worker-3   <none>           <none>
pod/rke2-ingress-nginx-controller-tj8hw                     1/1     Running     0               4d4h   10.42.xxx.xxx  inst-k8s-worker-5   <none>           <none>

service/rke2-ingress-nginx-controller-admission         ClusterIP   10.43.xxx.xxx    <none>        443/TCP                        97d    app.kubernetes.io/component=controller,app.kubernetes.io/instance=rke2-ingress-nginx,app.kubernetes.io/name=rke2-ingress-nginx
service/rke2-ingress-nginx-controller-metrics           ClusterIP   10.43.xxx.xxx    <none>        10254/TCP                      43d    app.kubernetes.io/component=controller,app.kubernetes.io/instance=rke2-ingress-nginx,app.kubernetes.io/name=rke2-ingress-nginx

daemonset.apps/rke2-ingress-nginx-controller        5         5         5       5            5           kubernetes.io/os=linux                   46d    rke2-ingress-nginx-controller                            rancher/nginx-ingress-controller:v1.12.0-hardened2                                                                                                                     app.kubernetes.io/component=controller,app.kubernetes.io/instance=rke2-ingress-nginx,app.kubernetes.io/name=rke2-ingress-nginx

job.batch/helm-install-rke2-ingress-nginx             Complete   1/1           98s        4d4h   helm         rancher/klipper-helm:v0.9.3-build20241008   batch.kubernetes.io/controller-uid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
$ kubectl -n kube-system describe pod/rke2-ingress-nginx-controller-blmrw
Name:             rke2-ingress-nginx-controller-blmrw
Namespace:        kube-system
Priority:         0
Service Account:  rke2-ingress-nginx
Node:             my-k8s-worker-n/10.xxx.xxx.xxx
Start Time:       Sun, 16 Feb 2025 12:29:41 +0100
Labels:           app.kubernetes.io/component=controller
                  app.kubernetes.io/instance=rke2-ingress-nginx
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=rke2-ingress-nginx
                  app.kubernetes.io/part-of=rke2-ingress-nginx
                  app.kubernetes.io/version=1.12.0
                  controller-revision-hash=6dd6ddb4bf
                  helm.sh/chart=rke2-ingress-nginx-4.12.003
                  pod-template-generation=7
Annotations:      prometheus.io/port: 10254
                  prometheus.io/scrape: true
Status:           Running
IP:               10.42.xxx.xxx
IPs:
  IP:           10.42.xxx.xxx
  IP:           fd00:42:xxxx:xxxx::xxxx
Controlled By:  DaemonSet/rke2-ingress-nginx-controller
Containers:
  rke2-ingress-nginx-controller:
    Container ID:    containerd://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    Image:           rancher/nginx-ingress-controller:v1.12.0-hardened2
    Image ID:        docker.io/rancher/nginx-ingress-controller@sha256:2607ab5b9d933fa887e44d4bfc329b1fb178f7ff08ec60a17e5dfb06917944ad
    Ports:           80/TCP, 443/TCP, 10254/TCP, 8443/TCP, 22/TCP, 4190/TCP, 465/TCP, 587/TCP, 993/TCP, 995/TCP
    Host Ports:      80/TCP, 443/TCP, 0/TCP, 0/TCP, 22/TCP, 4190/TCP, 465/TCP, 587/TCP, 993/TCP, 995/TCP
    SeccompProfile:  RuntimeDefault
    Args:
      /nginx-ingress-controller
      --election-id=rke2-ingress-nginx-leader
      --controller-class=k8s.io/ingress-nginx
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/rke2-ingress-nginx-controller
      --tcp-services-configmap=$(POD_NAMESPACE)/rke2-ingress-nginx-tcp
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
      --watch-ingress-without-class=true
      --enable-metrics=true
    State:          Running
      Started:      Sun, 16 Feb 2025 12:30:34 +0100
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       rke2-ingress-nginx-controller-blmrw (v1:metadata.name)
      POD_NAMESPACE:  kube-system (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rsnp4 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  rke2-ingress-nginx-admission
    Optional:    false
  kube-api-access-rsnp4:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:                      <none>
$ kubectl describe ingress myingress -n mynamespace
name:             myingress
Labels:           app.kubernetes.io/instance=myproject
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=myapp
                  app.kubernetes.io/version=0.9.9
                  argocd.argoproj.io/instance=myproject
                  helm.sh/chart=myapp-0.9.9
Namespace:        mynamespace
Address:          10.xxx.xxx.xxx,10.xxx.xxx.xxx,10.xxx.xxx.xxx,10.xxx.xxx.xxx,10.xxx.xxx.xxx
Ingress Class:    nginx
Default backend:  <default>
Rules:
  Host                     Path  Backends
  ----                     ----  --------
  www.example.com  
                           /   myproject-nginx:80 (10.42.xxx.xxx:80)
Annotations:               nginx.ingress.kubernetes.io/force-ssl-redirect: true
                           nginx.ingress.kubernetes.io/from-to-www-redirect: true
Events:                    <none>
  • Others:

N/A

How to reproduce this issue:

Setup the infrastructure

  • Create one or more instances

  • Create a loadbalancer

  • Generate a SSL certificate

  • Create a listener on port 443 with TLS Termination, all X-Forwarded headers enabled, and attach the SSL certificate

  • Create a pool with Proxy Protocol V2 backend

  • Add your nodes in the pool and output trafic on port 80

Install minikube/kind

Install the ingress controller

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/baremetal/deploy.yaml

Install an application that will act as default backend

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/http-svc.yaml

Create an ingress

echo "
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: foo-bar
    annotations:
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/force-ssl-redirect: true
      nginx.ingress.kubernetes.io/from-to-www-redirect: true
  spec:
    ingressClassName: nginx # omit this if you're on controller version below 1.0.0
    rules:
    - host: foo.bar
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: http-svc
              port: 
                number: 80
" | kubectl apply -f -
  • make a request from the outside

$ curl -kv https://www.foo.bar

Anything else we need to know:

That's all, I'm still available for more informations and debug, thank you for help :)

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.lifecycle/frozenIndicates that an issue or PR should not be auto-closed due to staleness.needs-priorityneeds-triageIndicates an issue or PR lacks a `triage/foo` label and requires one.

    Type

    No type

    Projects

    • Status

      No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions