Cert-manager output changed, causing constant reboots

[applejag]

Report

Cert-manager made a tiny change to the way they format their secrets.

In ca.crt field, instead of <PEM block> + <LF> + <LF>, they changed it to only have 1 trailing newline: <PEM block> + <LF>

This change was applied to the new mycluster-ssl and mycluster-ssl-internal secrets, but not yet in mycluster-ca-cert as it has not rotated yet.

So even though the two certs were the same, the following line thought the CA had been updated:

percona-server-mongodb-operator/pkg/controller/perconaservermongodb/ssl.go

Lines 241 to 243 in 923e4b3

	if bytes.Equal(secret.Data["ca.crt"], caSecret.Data["ca.crt"]) {
	continue
	}

The PSMDB Operator then deletes the mycluster-ssl and mycluster-ssl-internal, letting cert-manager regenerate them, but still with the new format of only 1 trailing newline. So next time PSMDB Operator checks, it repeats.

More about the problem

I tried applying a change to our PSMDB cluster, but the operator just kept on restarting the pods in the cluster, while complaining that:

ERROR	Reconciler error	{..., "error": "reconcile statefulsets: reconcile replset cfg: reconcile StatefulSet for cfg: get StatefulSet for replset cfg: failed to get ssl annotations: waiting for TLS secret", ...}

It took me a while before I finally saw the real issue in the logs:

INFO	createSSLByCertManager	CA is not up to date. Recreating secret

I suggest, if possible, to wait a little longer before logging that big waiting for TLS secret error message. Cert-manager creates the secret, and milliseconds later the operator sends 3 huge log messages that the secret is not ready yet. Generating self-signed certs is fast, so would be nice with something like a 1s timeout when waiting.

Steps to reproduce

1. Use an older cert-manager version. Don't know exactly, but v1.20.0 should have the "double newline" behavior
2. Install PSMDB cluster
3. Update cert-manager to something like v1.21.0, which only has the single trailing newline
4. Delete one of the mycluster-ssl or mycluster-ssl-internal secrets, so that cert-manager regenerates them with the new single-trailing-newline format
5. PSMDB operator gets into restart frenzy

Versions

1. Kubernetes v1.35.5
2. Operator v1.23.0
3. Database 8.0.26-11

Anything else?

Can be hotfixed by editing the mycluster-ca-cert to have the same ca.crt as mycluster-ssl. But would be nice if the PSMDB Operator did a better comparison, such as by parsing the two PEM blocks and comparing the parsed *x509.Certificate. Or at least add in a bytes.TrimSpace

if bytes.Equal(bytes.TrimSpace(secret.Data["ca.crt"]), bytes.TrimSpace(caSecret.Data["ca.crt"])) {

[egegunes]

@applejag thank you for the report. I created https://perconadev.atlassian.net/browse/K8SPSMDB-1789 to fix this and it's scheduled to next release (v1.24.0).