Skip to content

Operator recreates StatefulSet when MongoDBCommunity CR is deleted with cascade=foreground #753

@vinod014

Description

@vinod014

What did you do to encounter the bug?
Steps to reproduce the behavior:

  1. Install mongodb-kubernetes helm chart
  2. Install sample MongoDBCommunity CR
  3. Wait for the MongoDB ReplicaSet to be in Running phase
  4. Delete the MongoDBCommunity CR with foreground cascade deletion:
    kubectl delete mdbc <cr-name> --cascade=foreground
  5. Observe the operator logs - the StatefulSet gets recreated during the deletion process

What did you expect?
When the MongoDBCommunity CR is being deleted (regardless of cascade mode), the operator should:

  1. Detect that the CR has a DeletionTimestamp set
  2. Skip the normal reconciliation logic
  3. Allow Kubernetes garbage collection to clean up owned resources
  4. Not attempt to recreate the StatefulSet or any other dependent resources

What happened instead?
When using --cascade=foreground:

  1. The CR is marked for deletion (DeletionTimestamp is set) but still exists in the API server
  2. Kubernetes deletes dependent resources (StatefulSet) first due to owner references with blockOwnerDeletion: true
  3. The deletion of the StatefulSet triggers a reconciliation event (because the controller watches owned StatefulSets via Owns(&appsv1.StatefulSet{}))
  4. The controller fetches the CR successfully (it still exists, just marked for deletion)
  5. The controller proceeds with normal reconciliation and recreates the StatefulSet
  6. This causes a loop where the StatefulSet keeps getting recreated until the CR is finally removed

Operator Information

  • Operator Version is quay.io/mongodb/mongodb-kubernetes:1.6.0
  • MongoDB Image used is quay.io/mongodb/mongodb-community-server:8.0.4-ubi8

Kubernetes Cluster Information

  • Distribution is RKE2
  • Version is v1.34.3+rke2r1
  • Image Registry location is quay.io/mongodb/

Additional context
The controller only handles the case where the CR is completely gone (IsNotFound), but does not check if the CR is in the process of being deleted (DeletionTimestamp != nil).
This is not just a cascade=foreground issue. The operator should properly handle CR deletion in all scenarios.

❯ k get mdbc
NAME                            PHASE     VERSION
mongodb-standalone-replicaset   Pending   8.0.4

Operator logs for reconcile loop post deletion of the CR with --cascade=foreground

{"level":"info","ts":1770378775.584676,"caller":"controllers/replica_set_controller.go:156","msg":"Reconciling MongoDB","ReplicaSet":"<namespace>/<mongodb-cr-name>"}
{"level":"info","ts":1770378775.5879467,"caller":"controllers/replica_set_controller.go:472","msg":"Create/Update operation succeeded","ReplicaSet":"<namespace>/<mongodb-cr-name>","operation":"updated"}
{"level":"info","ts":1770378775.5881884,"caller":"controllers/replica_set_controller.go:364","msg":"Creating/Updating AutomationConfig","ReplicaSet":"<namespace>/<mongodb-cr-name>"}
{"level":"error","ts":1770378775.615704,"caller":"controllers/mongodb_status_options.go:104","msg":"Error deploying MongoDB ReplicaSet: failed to ensure AutomationConfig: could not build automation config: could not configure scram authentication: could not convert users to Automation Config users: failed to convert scram user <user-1> to Automation Config user: could not ensure scram credentials: could not determine if new credentials need to be generated: could not get secret <namespace>/<user-1-scram-credentials>: Secret \"<user-1-scram-credentials>\" not found","stacktrace":"..."}
{"level":"info","ts":1770378775.6309543,"caller":"controllers/replica_set_controller.go:156","msg":"Reconciling MongoDB","ReplicaSet":"<namespace>/<mongodb-cr-name>"}
{"level":"error","ts":1770378775.6346438,"caller":"controllers/replica_set_controller.go:468","msg":"Could not create or patch the service: Operation cannot be fulfilled on services \"<mongodb-cr-name>-svc\": the object has been modified; please apply your changes to the latest version and try again","ReplicaSet":"<namespace>/<mongodb-cr-name>","stacktrace":"..."}
{"level":"info","ts":1770378775.6349006,"caller":"controllers/replica_set_controller.go:364","msg":"Creating/Updating AutomationConfig","ReplicaSet":"<namespace>/<mongodb-cr-name>"}
{"level":"error","ts":1770378775.7355373,"caller":"controllers/mongodb_status_options.go:104","msg":"Error deploying MongoDB ReplicaSet: failed to ensure AutomationConfig: could not build automation config: could not configure scram authentication: could not convert users to Automation Config users: failed to convert scram user <user-2> to Automation Config user: could not ensure scram credentials: could not determine if new credentials need to be generated: could not get secret <namespace>/<user-2-scram-credentials>: Secret \"<user-2-scram-credentials>\" not found","stacktrace":"..."}

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions