psmdb-operator pod crashloop after the upgrade

[jurim76]

Report

After upgrading k8s cluster to version 1.31.6 and psmdb operator to version 1.19.1 (via helm chart), the operator's pod is in a crashloop state.

More about the problem

Pod's log contains the following error

2025-02-25T21:46:44.338Z        INFO    setup   Manager starting up     {"gitCommit": "ecf1a1457e1d3c44f77b2e58c4f57ac0d3dfb94e", "gitBranch": "release-1-19-1", "goVersion": "go1.23.6", "os": "linux", "arch": "amd64"}
2025-02-25T21:46:44.371Z        ERROR   setup   failed to register multicluster service {"error": "get api groups and resources: unable to retrieve the complete list of server APIs: external.metrics.k8s.io/v1beta1: stale GroupVersion discovery: external.metrics.k8s.io/v1beta1", "errorVerbose": "unable to retrieve the complete list of server APIs: external.metrics.k8s.io/v1beta1: stale GroupVersion discovery: external.metrics.k8s.io/v1beta1\nget api groups and resources\ngithub.com/percona/percona-server-mongodb-operator/pkg/mcs.Register\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/mcs/register.go:41\nmain.main\n\t/go/src/github.com/percona/percona-server-mongodb-operator/cmd/manager/main.go:132\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:272\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"}
main.main
        /go/src/github.com/percona/percona-server-mongodb-operator/cmd/manager/main.go:133
runtime.main
        /usr/local/go/src/runtime/proc.go:272

I don't use metrics-server in k8s cluster and never seen this error for a previous psmdb operator versions.

Steps to reproduce

1. Upgrade psmdb-operator helm chart to version 1.19.1
2. Check the operator's pod status

Versions

1. Kubernetes 1.31.6
2. Operator 1.19.1
3. Database 6.0.19-16

Anything else?

No response

[jurim76]

UPD

Operator fails when monitors all namespaces and keda-operator-metrics-apiserver is running in a cluster.

[hors]

Hi @jurim76 , please try to check main branch (but not for production needs). PSMDBO 1.19 does not support ks8 1.31.

[Girraj1997gg]

I have 2 clusters with gke version: 1.32.2-gke.1182003
Operator: 1.19.1

I am able to run percona-operator on dev cluster but not on prod cluster. Not sure what is the issue

[efeboyaci]

In my case, it wasn’t external.metrics.k8s.io, it was ext.cattle.io but i think the cause is the same, a stale apiservice. I was able to fix it by finding and deleting it.

kubectl get apiservice | grep ext.cattle.io
kubectl delete apiservice v1.ext.cattle.io

However I think this is just a patchwork fix, the operator shouldn’t crash just because an unrelated apiservice is unavailable.

[hors]

In my case, it wasn’t external.metrics.k8s.io, it was ext.cattle.io but i think the cause is the same, a stale apiservice. I was able to fix it by finding and deleting it.

kubectl get apiservice | grep ext.cattle.io
kubectl delete apiservice v1.ext.cattle.io
However I think this is just a patchwork fix, the operator shouldn’t crash just because an unrelated apiservice is unavailable.

Do you use PSMDB 1.19 or 1.20?

[efeboyaci]

In my case, it wasn’t external.metrics.k8s.io, it was ext.cattle.io but i think the cause is the same, a stale apiservice. I was able to fix it by finding and deleting it.
kubectl get apiservice | grep ext.cattle.io
kubectl delete apiservice v1.ext.cattle.io
However I think this is just a patchwork fix, the operator shouldn’t crash just because an unrelated apiservice is unavailable.

Do you use PSMDB 1.19 or 1.20?

I was using PSMDB operator 1.19.1 and thought updating might help, but it was still crashing with 1.20.1 as well. The issue might have started after the cluster kubernetes version was upgraded from 1.28.9 to 1.30.1.

[hors]

@jurim76 @efeboyaci @Girraj1997gg this issue will be fixed under https://perconadev.atlassian.net/browse/K8SPSMDB-1466 task