error condition (ErrorReconcile) is never cleared after transient sharding-transition error

[slackshadow]

Report

During a sharding transition (enabling sharding, or resizing spec.sharding.configsvrReplSet), a transient reconcile error is written into the CR status as error=True (Reason=ErrorReconcile). After the cluster recovers (ready=True, all pods Running, subsequent reconciles succeed), the error condition is never cleared: error=True and ready=True coexist indefinitely.

Example error message captured in the stale condition:

dial: ping mongo: connection() error occurred during connection handshake:
failed to connect to test-cluster-rs0-0.test-cluster-rs0.acto-namespace.svc.cluster.local:27017:
dial tcp: lookup ... on 10.200.0.10:53: no such host

The no such host error is a normal transient during pod recreation (DNS not yet registered); the cluster recovers moments later, but the status keeps reporting an error.

More about the problem

Status is permanently inconsistent with reality. In our observation the cluster reached ready=True, all pods Running, and the operator had zero Reconciler error entries for the last ~10 minutes — yet error=True (ErrorReconcile) remained set.

Root cause — pkg/controller/perconaservermongodb/status.go, updateStatus():

• On reconcileErr != nil, the error path unconditionally adds ClusterCondition{Type: AppStateError, Reason: "ErrorReconcile"} and returns early.
• The success path (reconcileErr == nil) never calls RemoveCondition(AppStateError) (the only RemoveCondition in the file targets ConditionTypePendingSmartUpdate).
• AddCondition only appends when no condition of the same Type exists; it never clears an existing one.

Why the existing fixes for related issues do not resolve this:

• Sharded cluster bootstrapping fails #2228 (sharded bootstrap race, fixed in commit 65f5c6f) addressed initialization failing (not primary). That fix prevents the cause of one class of transient error but does not change the fact that any transient error — including no such host during a legitimate transition — is permanently persisted. Even with the race fixed, other transient conditions (DNS, network, pod recreation) will still poison the status.
• K8SPSMDB-1429: improve cluster status #2148 / K8SPSMDB-1429 (improve cluster status) changed when ready is set (wait for all replica sets to join the shard). It does not touch error-condition lifecycle.
• The defect is in the status state machine: error conditions need to be cleared (or re-evaluated) on successful reconciliation, like the other conditions are.

Steps to reproduce

1. Deploy the operator and a non-sharded PerconaServerMongoDB (replset rs0, clusterRole: configsvr), wait for state=ready.
2. Enable sharding in one update:

kubectl patch psmdb test-cluster -n acto-namespace --type merge \
  -p '{"spec":{"sharding":{"enabled":true,"configsvrReplSet":{"size":3,"volumeSpec":{"persistentVolumeClaim":{"resources":{"requests":{"storage":"1Gi"}}}}},"mongos":{"size":1}}}}'

3. During the transition, observe error=True (ErrorReconcile) appear.
4. Wait until state=ready, all pods (cfg/rs0/mongos) Running, and Reconciler error disappears from the operator log.
5. Check the conditions:

kubectl get psmdb test-cluster -n acto-namespace \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status} ({.reason}){"\n"}{end}'
# -> error=True (ErrorReconcile) persists alongside ready=True

Also reproducible by scaling configsvrReplSet.size 3 -> 1 on a sharded cluster (same transient DNS error is persisted).

Versions

1. Kubernetes: v1.23.0
2. Database: Percona Server for MongoDB 8.0

Anything else?

Suggested fix direction: on the success path of updateStatus(), call RemoveCondition(AppStateError) (or reset the error condition) once the cluster state is not AppStateError, mirroring how transient conditions are handled elsewhere. A defensive alternative: only write the ErrorReconcile condition when the error has persisted across N consecutive reconciles.

[egegunes]

This will be fixed under https://perconadev.atlassian.net/browse/K8SPSMDB-1785

[slackshadow]

This will be fixed under https://perconadev.atlassian.net/browse/K8SPSMDB-1785

Thanks for the quick response :)