-
Notifications
You must be signed in to change notification settings - Fork 42
Open
Description
What did you do to encounter the bug?
Steps to reproduce the behavior:
- Install mongodb-kubernetes helm chart
- Install sample MongoDBCommunity CR
- Wait for the MongoDB ReplicaSet to be in
Runningphase - Delete the MongoDBCommunity CR with foreground cascade deletion:
kubectl delete mdbc <cr-name> --cascade=foreground
- Observe the operator logs - the StatefulSet gets recreated during the deletion process
What did you expect?
When the MongoDBCommunity CR is being deleted (regardless of cascade mode), the operator should:
- Detect that the CR has a
DeletionTimestampset - Skip the normal reconciliation logic
- Allow Kubernetes garbage collection to clean up owned resources
- Not attempt to recreate the StatefulSet or any other dependent resources
What happened instead?
When using --cascade=foreground:
- The CR is marked for deletion (
DeletionTimestampis set) but still exists in the API server - Kubernetes deletes dependent resources (StatefulSet) first due to owner references with
blockOwnerDeletion: true - The deletion of the StatefulSet triggers a reconciliation event (because the controller watches owned StatefulSets via
Owns(&appsv1.StatefulSet{})) - The controller fetches the CR successfully (it still exists, just marked for deletion)
- The controller proceeds with normal reconciliation and recreates the StatefulSet
- This causes a loop where the StatefulSet keeps getting recreated until the CR is finally removed
Operator Information
- Operator Version is quay.io/mongodb/mongodb-kubernetes:1.6.0
- MongoDB Image used is quay.io/mongodb/mongodb-community-server:8.0.4-ubi8
Kubernetes Cluster Information
- Distribution is RKE2
- Version is v1.34.3+rke2r1
- Image Registry location is quay.io/mongodb/
Additional context
The controller only handles the case where the CR is completely gone (IsNotFound), but does not check if the CR is in the process of being deleted (DeletionTimestamp != nil).
This is not just a cascade=foreground issue. The operator should properly handle CR deletion in all scenarios.
❯ k get mdbc
NAME PHASE VERSION
mongodb-standalone-replicaset Pending 8.0.4
Operator logs for reconcile loop post deletion of the CR with --cascade=foreground
{"level":"info","ts":1770378775.584676,"caller":"controllers/replica_set_controller.go:156","msg":"Reconciling MongoDB","ReplicaSet":"<namespace>/<mongodb-cr-name>"}
{"level":"info","ts":1770378775.5879467,"caller":"controllers/replica_set_controller.go:472","msg":"Create/Update operation succeeded","ReplicaSet":"<namespace>/<mongodb-cr-name>","operation":"updated"}
{"level":"info","ts":1770378775.5881884,"caller":"controllers/replica_set_controller.go:364","msg":"Creating/Updating AutomationConfig","ReplicaSet":"<namespace>/<mongodb-cr-name>"}
{"level":"error","ts":1770378775.615704,"caller":"controllers/mongodb_status_options.go:104","msg":"Error deploying MongoDB ReplicaSet: failed to ensure AutomationConfig: could not build automation config: could not configure scram authentication: could not convert users to Automation Config users: failed to convert scram user <user-1> to Automation Config user: could not ensure scram credentials: could not determine if new credentials need to be generated: could not get secret <namespace>/<user-1-scram-credentials>: Secret \"<user-1-scram-credentials>\" not found","stacktrace":"..."}
{"level":"info","ts":1770378775.6309543,"caller":"controllers/replica_set_controller.go:156","msg":"Reconciling MongoDB","ReplicaSet":"<namespace>/<mongodb-cr-name>"}
{"level":"error","ts":1770378775.6346438,"caller":"controllers/replica_set_controller.go:468","msg":"Could not create or patch the service: Operation cannot be fulfilled on services \"<mongodb-cr-name>-svc\": the object has been modified; please apply your changes to the latest version and try again","ReplicaSet":"<namespace>/<mongodb-cr-name>","stacktrace":"..."}
{"level":"info","ts":1770378775.6349006,"caller":"controllers/replica_set_controller.go:364","msg":"Creating/Updating AutomationConfig","ReplicaSet":"<namespace>/<mongodb-cr-name>"}
{"level":"error","ts":1770378775.7355373,"caller":"controllers/mongodb_status_options.go:104","msg":"Error deploying MongoDB ReplicaSet: failed to ensure AutomationConfig: could not build automation config: could not configure scram authentication: could not convert users to Automation Config users: failed to convert scram user <user-2> to Automation Config user: could not ensure scram credentials: could not determine if new credentials need to be generated: could not get secret <namespace>/<user-2-scram-credentials>: Secret \"<user-2-scram-credentials>\" not found","stacktrace":"..."}
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels