Reconciliation calls kube-apiserver 1000/sec with high concurrency and many databases

[jimethn]

Report

We have 100 PSMDB defined. The operator is calling kube-apiserver over 1000x/sec with a MAX_CONCURRENT_RECONCILES of 60. Reducing to 2 lowers it to merely dozens of calls per second.

More about the problem

We have EKS audit logging enabled and the increased concurrency is causing a noticeable increase to our cloudwatch bill.

Steps to reproduce

1. Be on operator 1.21.1
2. Turn up MAX_CONCURRENT_RECONCILES
3. Define many clusters

Versions

1. Kubernetes v1.32
2. Operator v1.21.1
3. Database 6-8 (various versions)

Anything else?

I reported it as a bug because I'm not sure this level of activity is intended. But maybe this is actually a feature request for a parameter to slow down the reconciliation loop.

[egegunes]

It's a known issue: https://perconadev.atlassian.net/browse/K8SPSMDB-650

@hors we might want to consider addressing this in next year.

[hors]

It's a known issue: https://perconadev.atlassian.net/browse/K8SPSMDB-650

@hors we might want to consider addressing this in next year.

Let's try to do it in PSMDBO 1.23.0