[Bug]: Updating certificate Secret don't update NGINX

[IdoHaritan]

Version

3.7.0

What Kubernetes platforms are you running on?

Openshift

Steps to reproduce

We have deployed the NGINX as usual, with a wildcard TLS Secret.

After updating the Secret in Oopen shift to a different certificate, NGINX doesn't update the wildcard certificate file in it and doesn't reload the service.
Which causes the NGINX to still work with the old certificate and not use the new one.

[github-actions]

Hi @IdoHaritan thanks for reporting!

Be sure to check out the docs and the Contributing Guidelines while you wait for a human to take a look at this 🙂

Cheers!

[AlexFenlon]

Hi @IdoHaritan, just a few questions

Are you using OSS or Plus?

Are you using NGINX Ingress Helm Operator?

Can you please provide your RBAC configuration.

Can you also attempt to upgrade to v4.0.0 if possible and that might solve your issue.

[IdoHaritan]

Hi @AlexFenlon,

Yes, we are using NGINX Plus.

We are not using the NGINX Ingress Helm Operator, we are installing manually.

Our RBAC configuration files are attached below.
RBAC.zip

Unfortunately, it is not possible for us to upgrade to v4.0.0.

[vepatel]

@IdoHaritan since you're using Nginx+, I'd recommend opening a ticket with F5!

[haywoodsh]

@IdoHaritan I'm trying to understand your deployment. Did you mean you deployed the Ingress Controller with the command line argument -wildcard-tls-secret?

Does NGINX pick up the new secret if you delete the pod and wait for a new pod to start? If so, I think the issue you're having is due to the Ingress Controller only processes the wildcard secret when it starts, but does not monitor further changes.

I can see from the codebase that the function processWildcardSecret is only called in main.go, similar to other secrets configurable in the command line argument.

	isWildcardEnabled, err := processWildcardSecret(kubeClient, nginxManager)

https://github.com/nginx/kubernetes-ingress/blob/main/cmd/nginx-ingress/main.go#L181

If you need to update the wildcard secret during Ingress Controller runtime, can you let us know your use case?

[IdoHaritan]

Hi @haywoodsh, thank you.
Our wildcard Secret is changing frequently.
This can happen for several reasons, either because the certificate has expired or because of changes in the certificate's SANs.
So we need to update it during Ingress Controller runtime.

[haywoodsh]

We've been looking into codebase, and there is indeed an integration test showing the Ingress Controller does process and save the secret when it updates.
https://github.com/nginx/kubernetes-ingress/blob/main/tests/suite/test_wildcard_tls_secret.py#L165-L188
Since the tests are passing, I'm afraid this could indeed be an OpenShift specific environment issue, like we previously suggested. I'll try and see if I can reproduce this on my end.

[haywoodsh]

Hi @IdoHaritan, Following up on this. I ran a test with v3.7.1 manually following our integration tests. While the Ingress Controller was running, I changed the content of the secret specified via wildcard-tls-secret during runtime, switching between the data in our two test files:

https://github.com/nginx/kubernetes-ingress/blob/main/tests/data/common-secrets/wildcard-tls-secret-example.com.yaml
https://github.com/nginx/kubernetes-ingress/blob/main/tests/data/common-secrets/wildcard-tls-secret-example.com-gb.yaml

The controller picked up this change successfully. nmap showed the served certificate changing accordingly without any pod restarts:

# before update
❯ nmap -p 443 --script ssl-cert cafe.example.com
Starting Nmap 7.95 ( https://nmap.org/ ) at 2025-04-15 16:16 IST
Nmap scan report for cafe.example.com (127.0.0.1)
Host is up (0.00014s latency).
rDNS record for 127.0.0.1: localhost

PORT    STATE SERVICE
443/tcp open  https
| ssl-cert: Subject: commonName=example.com/organizationName=nginx/stateOrProvinceName=CanaryIslands/countryName=ES
| Issuer: commonName=example.com/organizationName=nginx/stateOrProvinceName=CanaryIslands/countryName=ES
| Public Key type: rsa
| Public Key bits: 2048
| Signature Algorithm: sha256WithRSAEncryption
| Not valid before: 2024-12-19T15:40:06
| Not valid after:  2034-12-17T15:40:06
| MD5:   56ce:6189:49cb:67f4:4a03:f16a:6c96:2048
|_SHA-1: 24d5:b77d:71c3:e8a0:ead7:63fe:02e7:a897:b06a:3aff

Nmap done: 1 IP address (1 host up) scanned in 0.08 seconds

# after update
❯ nmap -p 443 --script ssl-cert cafe.example.com
Starting Nmap 7.95 ( https://nmap.org/ ) at 2025-04-15 16:18 IST
Nmap scan report for cafe.example.com (127.0.0.1)
Host is up (0.00018s latency).
rDNS record for 127.0.0.1: localhost

PORT    STATE SERVICE
443/tcp open  https
| ssl-cert: Subject: commonName=example.com/organizationName=nginx/stateOrProvinceName=Cambridgeshire/countryName=GB
| Issuer: commonName=example.com/organizationName=nginx/stateOrProvinceName=Cambridgeshire/countryName=GB
| Public Key type: rsa
| Public Key bits: 2048
| Signature Algorithm: sha256WithRSAEncryption
| Not valid before: 2024-12-19T15:41:33
| Not valid after:  2034-12-17T15:41:33
| MD5:   3c0e:cd3e:b86d:2b72:8250:d40c:f749:7905
|_SHA-1: 37c3:cb25:7f51:3dfe:2d86:628f:518d:9e80:7976:fc3e

Nmap done: 1 IP address (1 host up) scanned in 0.08 seconds

When this update works, the Ingress Controller logs an event like this:

2025-04-15T18:44:49.922918639Z I20250415 18:44:49.922824       1 event.go:377] Event(v1.ObjectReference{Kind:"Secret", Namespace:"nginx-ingress", Name:"wildcard-secret", UID:"5290de10-989c-4cc4-ad01-a26327e0c168", APIVersion:"v1", ResourceVersion:"1137530", FieldPath:""}): type: 'Normal' reason: 'Updated' the special Secret nginx-ingress/wildcard-secret was updated

This confirms the Ingress Controller can process runtime updates to the wildcard secret, which relies on the controller actively watching the namespace where the secret resides.

A scenario where the Ingress Controller wouldn't see or process updates after starting is when the command-line argument --watch-secret-namespace or --watch-namespace are set to exclude the wildcard secret's namespace. Please make sure these two arguments are either unset or include your wildcard secret's namespace.

Given that the integration passes and this manual checking worked under standard configuration, it seems that there are some environment specific issues in your cluster causing the Ingress Controller not watching the secret as intended.

[IdoHaritan]

Hi @haywoodsh

A scenario where the Ingress Controller wouldn't see or process updates after starting is when the command-line argument --watch-secret-namespace or --watch-namespace are set to exclude the wildcard secret's namespace. Please make sure these two arguments are either unset or include your wildcard secret's namespace.

We are indeed using one of the arguments above in our configuration.

In our environment, we have deployed the Ingress Controller on a namespace and used the argument --watch-namespace-label to watch only namespaces with a certain label defined.
So the Ingress Controller is watching for VirtualServers in multiple different namespaces (other than the namespace it deployed on).

We would like to avoid adding this label to the Ingress Controller namespace.

---

1. If we will add the argument --watch-secret-namespace, it will require us to add the label to the Ingress Controller namespace?
2. What is your recommendation for our scenario?

Thanks