cert-manager renewal and refreshing the NMA

[matchaxnb]

Hello Vertica folks!

This applies to operator version 25.4.0 (the patched one) but could very well apply to more recent versions.

TL;DR

Cert-manager node management certificate rotation causes issues between the NMA and operator. This can happen during the renewal period if any Vertica node or the operator itself restarts post-renewal and picks up the new certificate while the other participants still use the old one.

Restarting the NMA processes so they use the up-to-date certificate and reloading the certificate in the operator is necessary for continued operation.

Until a better fix is provided, we have a mitigation in the form of a shell script that we can run to restore certificate alignment.

My solution to restore functionality is to kill -HUP the nma on all pods and to reschedule the operator itself so it loads the new certificate using the k8s API. Then the operator is able to speak to the NMA and to reconcile the cluster.

Steps to reproduce

• Use VerticaDB operator and deploy a VerticaDB, using an externally managed cert-manager certificate
• Rotate the certificate using cmctl renew or whatever means available, so that there is an heterogeneity of node-management certificate between all participants in the cluster, like the different nodes and the operator. This will become manifest as soon as one of the nodes restart for example, or if the operator itself restarts.
• Restart the operator so that it picks up the new certificate
• Disrupt one of the nodes, such as by killing its Vertica process, to simulate a failure: the operator will not be able to act on it and will frantically log. At this point, although the operator and the new node have the same certificate, because the operator and the other nodes are not aligned certificate-wise, the operations will fail
• Restart the NMA on all nodes by issuing a kill -HUP 1 in the nma containers: once this is done, the certificate will be aligned again and operations will work.

Why it is a problem

Using an external certificate is supported by Vertica, as we can provide a nmaTLSSecret key for that purpose. Therefore, the NMA and operator should be able to reload the certificate live.

My script

restore_vertica_operator() {
_cluster="$1"
shift 1
echo "Restore the NMA + operator functionality"
echo "First, kill the NMA processes"
kubectl -n vertica get pods -l app.kubernetes.io/instance=$_cluster -o name | while read podname; do
  kubectl -n vertica exec -c nma $podname -- kill -HUP 1 &
done
wait # Wait for the background processes to complete, i.e. for the NMAs to be killed
echo "now restart the operator"
kubectl -n vertica delete pod -l app.kubernetes.io/name=verticadb-operator
echo "Done!"
}
# example usage: restore_vertica_operator multitenant

If needed, I can reproduce the issue in full with timestamps and show an example, but I think it's pretty straightforward to reproduce.

[roypaulin]

@matchaxnb If I am not wrong, you are referring to an in-place update of the secret. controller-manager drives the rotation and the existing secret is updated.
We don't recommend and don't support in-place update. The way we do cert rotation is by creating/providing a new secret with new cert. That will trigger a reconciliation loop and the operator itself will drive cert rotation.