CA private key rotation deadlocks the TLS rolling update

[pehlicd]

Report

The operator creates the <cluster>-ca-cert Certificate without
spec.privateKey, so it inherits the cert-manager default. Since cert-manager
v1.18.0 that default is rotationPolicy: Always, so every CA renewal also
generates a new CA private key. When that happens the cluster does not survive the
rolling update.

More about the problem

On CA renewal the first restarted mongos cannot reach the mongod pods that have
not restarted yet. It enters CrashLoopBackOff and the rollout stalls with part
of the cluster on the old CA and part on the new one. The operator logs:

createSSLByCertManager: CA is not up to date. Recreating secret

Expected behavior: the cluster rolls through a CA renewal without downtime, which
is what mergeNewCA and the mongos-first restart order are designed to achieve.

Steps to reproduce

1. Install cert-manager v1.21.0 or newer.
2. Deploy a sharded cluster with TLS enabled and certificates created by the
   operator via cert-manager.
3. Renew the CA certificate: cmctl renew <cluster>-ca-cert, or wait until
   renewBefore is reached.
4. Watch the mongos and mongod pods roll.

Versions

1. Kubernetes 1.35.5, on premise
2. Operator 1.23.0
3. Percona Server for MongoDB 7.0.28-15

Anything else?

cert-manager 1.21.0. The default changed from Never to Always in cert-manager
v1.18.0 and went GA in v1.19.0, where the feature gate can no longer be disabled.

See: cert-manager/cert-manager#7601

We have hit this three times since March 2026 on three different clusters, each
time needing manual intervention on a running cluster to recover with a downtime.

[egegunes]

I created https://perconadev.atlassian.net/browse/K8SPSMDB-1792 to fix this.