Description
What happened:
When using the annotation nginx.ingress.kubernetes.io/from-to-www-redirect: 'true'
on any of my Nginx ingresses, requests to https://www.example.com is incorrectly redirected to https://example.com:80, causing an access error (trying to access using TLS on plaintext HTTP port).
Behind my Kubernetes cluster, there is a Loadbalancer with two listeners.
One of them, listening on port 443, has TLS Terminating / TLS Offloading feature (it has direct access to the SSL certificates).
The Loadbalancer is setup on Openstack Octavia and is using HAProxy.
Recap :
Public trafic comes on port 443 of the LB.
Request is uncrypted then forwarded to the :80 pool.
Standard clear requests on port 80 are also forwarded to the :80 pool.
On every request, the following HTTP headers are inserted by the LB listeners :
- X-Forwarded-For
- X-Forwarded-Port
- X-Forwarded-Proto
(HTTP plaintext request)
Client -----> Listener :80 ------------> Pool :80 ------ Proxy Protocol V2 -----> K8S Worker Nodes :80
/
TLS Termination
(HTTPS encrypted request) /
Client -----> Listener :443 /
This issue persists even with the following configuration :
controller:
config:
enable-real-ip: "true"
proxy-real-ip-cidr: 10.xxx.xxx.0/24,2001:xxxx:xxxx:106::/64
use-forwarded-headers: "true"
use-proxy-protocol: "true"
service:
targetPorts:
http: http
https: http
What you expected to happen:
Nginx Ingress Controller should take care of X-Forwarded-Port
and X-Forwarded-Proto
, to know the request was originally coming from port 443 and was HTTPS.
It should instead redirect https://www.example.com to https://example.com:443 (or simply https://example.com).
NGINX Ingress controller version :
www-data@rke2-ingress-nginx-controller-s5b5j:/etc/nginx> /nginx-ingress-controller --version
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: v1.12.0-hardened1
Build: git-5bfb8f4b7
Repository: https://github.com/rancher/ingress-nginx
nginx version: nginx/1.25.5
-------------------------------------------------------------------------------
Kubernetes version : v1.31.5+rke2r1
Environment:
-
Cloud provider or hardware configuration : OpenStack (hosted by Infomaniak.com)
-
OS : Ubuntu 24.04.2 LTS
-
Kernel :
6.8.0-51-generic
-
Install tools:
- Infrastructure was setup using OpenTofu + CloudInit for nodes
- Cluster was setup using Ansible (thanks to Labyrinth Labs)
-
Basic cluster related info:
- 3 Master nodes (with taint NoExecute)
- 6 Worker nodes
- Cilium CNI (eBPF routing, Geneve tunneling, kube-proxy replacement enabled, DSR enabled, XDP enabled)
- IPv4 / IPv6 Dualstack (fully working)
-
How was the ingress-nginx-controller installed:
$ helm ls -A | grep -i ingress
rke2-ingress-nginx kube-system 101 2025-02-20 03:52:46.70060196 +0100 CET deployed rke2-ingress-nginx-4.12.003 1.12.0
- Current State of the controller:
$ kubectl describe ingressclasses
Name: nginx
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=rke2-ingress-nginx
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=rke2-ingress-nginx
app.kubernetes.io/part-of=rke2-ingress-nginx
app.kubernetes.io/version=1.12.0
helm.sh/chart=rke2-ingress-nginx-4.12.003
Annotations: meta.helm.sh/release-name: rke2-ingress-nginx
meta.helm.sh/release-namespace: kube-system
Controller: k8s.io/ingress-nginx
Events: <none>
$ kubectl -n kube-system get all -o wide | grep ingress
pod/helm-install-rke2-ingress-nginx-k7p56 0/1 Completed 0 4d4h <none> inst-k8s-worker-3 <none> <none>
pod/rke2-ingress-nginx-controller-blmrw 1/1 Running 0 4d4h 10.42.xxx.xxx inst-k8s-worker-2 <none> <none>
pod/rke2-ingress-nginx-controller-j67b9 1/1 Running 0 4d4h 10.42.xxx.xxx inst-k8s-worker-1 <none> <none>
pod/rke2-ingress-nginx-controller-l22rl 1/1 Running 0 4d4h 10.42.xxx.xxx inst-k8s-worker-4 <none> <none>
pod/rke2-ingress-nginx-controller-s5b5j 1/1 Running 0 4d4h 10.42.xxx.xxx inst-k8s-worker-3 <none> <none>
pod/rke2-ingress-nginx-controller-tj8hw 1/1 Running 0 4d4h 10.42.xxx.xxx inst-k8s-worker-5 <none> <none>
service/rke2-ingress-nginx-controller-admission ClusterIP 10.43.xxx.xxx <none> 443/TCP 97d app.kubernetes.io/component=controller,app.kubernetes.io/instance=rke2-ingress-nginx,app.kubernetes.io/name=rke2-ingress-nginx
service/rke2-ingress-nginx-controller-metrics ClusterIP 10.43.xxx.xxx <none> 10254/TCP 43d app.kubernetes.io/component=controller,app.kubernetes.io/instance=rke2-ingress-nginx,app.kubernetes.io/name=rke2-ingress-nginx
daemonset.apps/rke2-ingress-nginx-controller 5 5 5 5 5 kubernetes.io/os=linux 46d rke2-ingress-nginx-controller rancher/nginx-ingress-controller:v1.12.0-hardened2 app.kubernetes.io/component=controller,app.kubernetes.io/instance=rke2-ingress-nginx,app.kubernetes.io/name=rke2-ingress-nginx
job.batch/helm-install-rke2-ingress-nginx Complete 1/1 98s 4d4h helm rancher/klipper-helm:v0.9.3-build20241008 batch.kubernetes.io/controller-uid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
$ kubectl -n kube-system describe pod/rke2-ingress-nginx-controller-blmrw
Name: rke2-ingress-nginx-controller-blmrw
Namespace: kube-system
Priority: 0
Service Account: rke2-ingress-nginx
Node: my-k8s-worker-n/10.xxx.xxx.xxx
Start Time: Sun, 16 Feb 2025 12:29:41 +0100
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=rke2-ingress-nginx
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=rke2-ingress-nginx
app.kubernetes.io/part-of=rke2-ingress-nginx
app.kubernetes.io/version=1.12.0
controller-revision-hash=6dd6ddb4bf
helm.sh/chart=rke2-ingress-nginx-4.12.003
pod-template-generation=7
Annotations: prometheus.io/port: 10254
prometheus.io/scrape: true
Status: Running
IP: 10.42.xxx.xxx
IPs:
IP: 10.42.xxx.xxx
IP: fd00:42:xxxx:xxxx::xxxx
Controlled By: DaemonSet/rke2-ingress-nginx-controller
Containers:
rke2-ingress-nginx-controller:
Container ID: containerd://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Image: rancher/nginx-ingress-controller:v1.12.0-hardened2
Image ID: docker.io/rancher/nginx-ingress-controller@sha256:2607ab5b9d933fa887e44d4bfc329b1fb178f7ff08ec60a17e5dfb06917944ad
Ports: 80/TCP, 443/TCP, 10254/TCP, 8443/TCP, 22/TCP, 4190/TCP, 465/TCP, 587/TCP, 993/TCP, 995/TCP
Host Ports: 80/TCP, 443/TCP, 0/TCP, 0/TCP, 22/TCP, 4190/TCP, 465/TCP, 587/TCP, 993/TCP, 995/TCP
SeccompProfile: RuntimeDefault
Args:
/nginx-ingress-controller
--election-id=rke2-ingress-nginx-leader
--controller-class=k8s.io/ingress-nginx
--ingress-class=nginx
--configmap=$(POD_NAMESPACE)/rke2-ingress-nginx-controller
--tcp-services-configmap=$(POD_NAMESPACE)/rke2-ingress-nginx-tcp
--validating-webhook=:8443
--validating-webhook-certificate=/usr/local/certificates/cert
--validating-webhook-key=/usr/local/certificates/key
--watch-ingress-without-class=true
--enable-metrics=true
State: Running
Started: Sun, 16 Feb 2025 12:30:34 +0100
Ready: True
Restart Count: 0
Requests:
cpu: 100m
memory: 90Mi
Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
Environment:
POD_NAME: rke2-ingress-nginx-controller-blmrw (v1:metadata.name)
POD_NAMESPACE: kube-system (v1:metadata.namespace)
LD_PRELOAD: /usr/local/lib/libmimalloc.so
Mounts:
/usr/local/certificates/ from webhook-cert (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rsnp4 (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
webhook-cert:
Type: Secret (a volume populated by a Secret)
SecretName: rke2-ingress-nginx-admission
Optional: false
kube-api-access-rsnp4:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events: <none>
$ kubectl describe ingress myingress -n mynamespace
name: myingress
Labels: app.kubernetes.io/instance=myproject
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=myapp
app.kubernetes.io/version=0.9.9
argocd.argoproj.io/instance=myproject
helm.sh/chart=myapp-0.9.9
Namespace: mynamespace
Address: 10.xxx.xxx.xxx,10.xxx.xxx.xxx,10.xxx.xxx.xxx,10.xxx.xxx.xxx,10.xxx.xxx.xxx
Ingress Class: nginx
Default backend: <default>
Rules:
Host Path Backends
---- ---- --------
www.example.com
/ myproject-nginx:80 (10.42.xxx.xxx:80)
Annotations: nginx.ingress.kubernetes.io/force-ssl-redirect: true
nginx.ingress.kubernetes.io/from-to-www-redirect: true
Events: <none>
- Others:
N/A
How to reproduce this issue:
Setup the infrastructure
-
Create one or more instances
-
Create a loadbalancer
-
Generate a SSL certificate
-
Create a listener on port 443 with TLS Termination, all X-Forwarded headers enabled, and attach the SSL certificate
-
Create a pool with Proxy Protocol V2 backend
-
Add your nodes in the pool and output trafic on port 80
Install minikube/kind
- Minikube https://minikube.sigs.k8s.io/docs/start/
- Kind https://kind.sigs.k8s.io/docs/user/quick-start/
Install the ingress controller
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/baremetal/deploy.yaml
Install an application that will act as default backend
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/docs/examples/http-svc.yaml
Create an ingress
echo "
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: foo-bar
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/force-ssl-redirect: true
nginx.ingress.kubernetes.io/from-to-www-redirect: true
spec:
ingressClassName: nginx # omit this if you're on controller version below 1.0.0
rules:
- host: foo.bar
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: http-svc
port:
number: 80
" | kubectl apply -f -
- make a request from the outside
$ curl -kv https://www.foo.bar
Anything else we need to know:
That's all, I'm still available for more informations and debug, thank you for help :)
Metadata
Metadata
Assignees
Labels
Type
Projects
Status
No status