Skip to content

MDBC - topologySpreadConstraints change from 2 to 1 does not trigger reconciliation #700

@ivanmartos

Description

@ivanmartos

What did you do to encounter the bug?

  • deploy mdbc custom resource with 2 spec.statefulSet.spec.template.spec.topologySpreadConstraints
  • wait for reconciliation
    • stateful set has correct number of topologySpreadConstraints
  • update mdbc custom resource to have just 1 spec.statefulSet.spec.template.spec.topologySpreadConstraints

What did you expect?
After reducing number of topologySpreadConstraints on mdbc resource from 2 to 1 I would expect that statefulset will have correct number of topologySpreadConstraints - 1

What happened instead?
After reducing topologySpreadConstraints from 2 to 1 reconciliation was not triggered and statefulSet has still 2 topologySpreadConstraints

Even when reconciliation was triggered after some time due to other reason, number of topologySpreadConstraints on statefulSet was still 2 instead of 1

Operator Information

  • Operator Version - mongodb/mongodb-kubernetes:1.6.0
  • MongoDB Image used - MongoDBCommunity CRD with mongodb/mongodb-community-server:8.2.2-ubi8

Kubernetes Cluster Information

  • Distribution - AKS
  • Version - 1.32.9
  • Image Registry location (quay, or an internal registry) - azure container registry

Additional context

% k get pods -n management -l app=management-application-mongodb-headless
NAME                               READY   STATUS    RESTARTS   AGE
management-application-mongodb-0   2/2     Running   0          6h27m
management-application-mongodb-1   2/2     Running   0          4m29s
management-application-mongodb-2   2/2     Running   0          3m42s
                                                                                     
% k get mdbc --all-namespaces
NAMESPACE    NAME                             PHASE     VERSION
management   management-application-mongodb   Running   8.2.2-ubi8

mdbc yaml

apiVersion: mongodbcommunity.mongodb.com/v1
kind: MongoDBCommunity
metadata:
  annotations:
    meta.helm.sh/release-name: example-application-mongodb-deployment
    meta.helm.sh/release-namespace: example
    mongodb.com/v1.lastAppliedMongoDBVersion: 8.2.2-ubi8
    mongodb.com/v1.lastSuccessfulConfiguration: 'REDACTED'
  creationTimestamp: "2025-12-19T10:06:10Z"
  generation: 4
  labels:
    app.kubernetes.io/managed-by: Helm
    helm.toolkit.fluxcd.io/name: example-application-mongodb-deployment
    helm.toolkit.fluxcd.io/namespace: flux-resources
  name: example-application-mongodb
  namespace: example
  resourceVersion: "REDACTED"
  uid: REDACTED
spec:
  featureCompatibilityVersion: "8.0"
  members: 3
  security:
    authentication:
      ignoreUnknownUsers: true
      modes:
        - SCRAM
  statefulSet:
    spec:
      minReadySeconds: 30
      serviceName: example-application-mongodb-headless
      template:
        metadata:
          labels:
            sidecar.istio.io/inject: "false"
            sidecar.gw.example.com/inject: "false"
        spec:
          containers:
            - livenessProbe:
                exec:
                  command:
                    - /bin/sh
                    - -c
                    - HOME=/tmp mongosh --quiet --norc --eval "db.adminCommand('ping')"
                failureThreshold: 3
                periodSeconds: 10
                timeoutSeconds: 5
              name: mongod
              resources:
                limits:
                  memory: 1500Mi
                requests:
                  cpu: 100m
                  memory: 1500Mi
              startupProbe:
                exec:
                  command:
                    - /bin/sh
                    - -c
                    - HOME=/tmp mongosh --quiet --norc --eval "db.adminCommand('ping')"
                failureThreshold: 30
                initialDelaySeconds: 5
                periodSeconds: 10
                timeoutSeconds: 5
            - name: mongodb-agent
              readinessProbe:
                failureThreshold: 60
                periodSeconds: 20
                timeoutSeconds: 10
              resources:
                limits:
                  memory: 130Mi
                requests:
                  cpu: 100m
                  memory: 130Mi
              startupProbe:
                exec:
                  command:
                    - curl
                    - -sf
                    - http://localhost:5000/healthz
                failureThreshold: 10
                initialDelaySeconds: 5
                periodSeconds: 10
                timeoutSeconds: 5
          nodeSelector:
            node.example.com/agentpool: worker
          priorityClassName: platform-system
          tolerations:
            - effect: PreferNoSchedule
              key: node.example.com/agentpool
              operator: Equal
              value: worker
            - effect: NoSchedule
              key: kubernetes.azure.com/scalesetpriority
              operator: Equal
              value: spot
          topologySpreadConstraints:
            - labelSelector:
                matchLabels:
                  app: example-application-mongodb-headless
              matchLabelKeys:
                - controller-revision-hash
              maxSkew: 1
              nodeTaintsPolicy: Honor
              topologyKey: kubernetes.io/hostname
              whenUnsatisfiable: DoNotSchedule
      volumeClaimTemplates:
        - apiVersion: v1
          kind: PersistentVolumeClaim
          metadata:
            name: data-volume
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 10G
            storageClassName: custom-retain
            volumeMode: Filesystem
        - apiVersion: v1
          kind: PersistentVolumeClaim
          metadata:
            name: logs-volume
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 2G
            storageClassName: custom-retain
            volumeMode: Filesystem
  type: ReplicaSet
  users:
    - db: example-cloud
      name: REDACTED
      passwordSecretRef:
        name: REDACTED
      roles:
        - db: example-cloud
          name: dbOwner
      scramCredentialsSecretName: example-center
    - db: admin
      name: REDACTED
      passwordSecretRef:
        name: REDACTED
      roles:
        - db: admin
          name: backup
        - db: admin
          name: restore
        - db: admin
          name: clusterMonitor
      scramCredentialsSecretName: mongodb-backup-restore
  version: 8.2.2-ubi8
status:
  currentMongoDBMembers: 3
  currentStatefulSetReplicas: 3
  mongoUri: REDACTED
  phase: Running
  version: 8.2.2-ubi8

stateful set created by operator

apiVersion: apps/v1
kind: StatefulSet
metadata:
  creationTimestamp: "2025-12-19T10:06:10Z"
  generation: 779
  name: example-application-mongodb
  namespace: example
  ownerReferences:
    - apiVersion: mongodbcommunity.mongodb.com/v1
      blockOwnerDeletion: true
      controller: true
      kind: MongoDBCommunity
      name: example-application-mongodb
      uid: REDACTED
  resourceVersion: "REDACTED"
  uid: REDACTED
spec:
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Retain
  podexamplePolicy: OrderedReady
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: example-application-mongodb-headless
  serviceName: example-application-mongodb-headless
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: example-application-mongodb-headless
        sidecar.istio.io/inject: "false"
        sidecar.gw.example.com/inject: "false"
    spec:
      containers:
        - args:
            - ""
          command:
            - /bin/sh
            - -c
            - "\nif [ -e \"/hooks/version-upgrade\" ]; then\n\t#run post-start hook to
          handle version changes (if exists)\n    /hooks/version-upgrade\nfi\n\n#
          wait for config and keyfile to be created by the agent\necho \"Waiting for
          config and keyfile files to be created by the agent...\"\nwhile ! [ -f /data/automation-mongod.conf
          -a -f /var/lib/mongodb-mms-automation/authentication/keyfile ]; do\n\tsleep
          3;\n\techo \"Waiting...\"\ndone\n\n# sleep is important after agent issues
          shutdown command\n# k8s restarts the mongod container too quickly for the
          agent to realize mongod is down\necho \"Sleeping for 15s...\"\nsleep 15\n\n#
          start mongod with this configuration\necho \"Starting mongod...\"\nexec
          mongod -f /data/automation-mongod.conf\n"
          env:
            - name: AGENT_STATUS_FILEPATH
              value: /healthstatus/agent-health-status.json
          image: examplesaas.azurecr.io/mongodb/mongodb-community-server:8.2.2-ubi8
          imagePullPolicy: IfNotPresent
          livenessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - HOME=/tmp mongosh --quiet --norc --eval "db.adminCommand('ping')"
            failureThreshold: 3
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 5
          name: mongod
          resources:
            limits:
              memory: 1500Mi
            requests:
              cpu: 100m
              memory: 1500Mi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          startupProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - HOME=/tmp mongosh --quiet --norc --eval "db.adminCommand('ping')"
            failureThreshold: 30
            initialDelaySeconds: 5
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 5
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /data
              name: data-volume
            - mountPath: /healthstatus
              name: healthstatus
            - mountPath: /hooks
              name: hooks
            - mountPath: /var/log/mongodb-mms-automation
              name: logs-volume
            - mountPath: /var/lib/mongodb-mms-automation/authentication
              name: example-application-mongodb-keyfile
            - mountPath: /tmp
              name: tmp
        - command:
            - /bin/bash
            - -c
            - |-
              current_uid=$(id -u)
              declare -r current_uid
              if ! grep -q "${current_uid}" /etc/passwd ; then
              sed -e "s/^mongodb:/builder:/" /etc/passwd > /tmp/passwd
              echo "mongodb:x:$(id -u):$(id -g):,,,:/:/bin/bash" >> /tmp/passwd
              export NSS_WRAPPER_PASSWD=/tmp/passwd
              export LD_PRELOAD=libnss_wrapper.so
              export NSS_WRAPPER_GROUP=/etc/group
              fi
              agent/mongodb-agent -healthCheckFilePath=/var/log/mongodb-mms-automation/healthstatus/agent-health-status.json -serveStatusPort=5000 -cluster=/var/lib/automation/config/cluster-config.json -skipMongoStart -noDaemonize -useLocalMongoDbTools -logFile /var/log/mongodb-mms-automation/automation-agent.log -logLevel INFO -maxLogFileDurationHrs 24
          env:
            - name: AGENT_STATUS_FILEPATH
              value: /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
            - name: AUTOMATION_CONFIG_MAP
              value: example-application-mongodb-config
            - name: HEADLESS_AGENT
              value: "true"
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
          image: examplesaas.azurecr.io/mongodb/mongodb-agent:108.0.2.8729-1
          imagePullPolicy: Always
          name: mongodb-agent
          readinessProbe:
            exec:
              command:
                - /opt/scripts/readinessprobe
            failureThreshold: 60
            initialDelaySeconds: 5
            periodSeconds: 20
            successThreshold: 1
            timeoutSeconds: 10
          resources:
            limits:
              memory: 130Mi
            requests:
              cpu: 100m
              memory: 130Mi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          startupProbe:
            exec:
              command:
                - curl
                - -sf
                - http://localhost:5000/healthz
            failureThreshold: 10
            initialDelaySeconds: 5
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 5
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /opt/scripts
              name: agent-scripts
            - mountPath: /var/lib/automation/config
              name: automation-config
              readOnly: true
            - mountPath: /data
              name: data-volume
            - mountPath: /var/log/mongodb-mms-automation/healthstatus
              name: healthstatus
            - mountPath: /var/log/mongodb-mms-automation
              name: logs-volume
            - mountPath: /var/lib/mongodb-mms-automation/authentication
              name: example-application-mongodb-keyfile
            - mountPath: /tmp
              name: tmp
      dnsPolicy: ClusterFirst
      initContainers:
        - command:
            - cp
            - version-upgrade-hook
            - /hooks/version-upgrade
          image: examplesaas.azurecr.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.10
          imagePullPolicy: Always
          name: mongod-posthook
          resources:
            limits:
              cpu: "1"
              memory: 500M
            requests:
              cpu: 500m
              memory: 400M
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /hooks
              name: hooks
        - command:
            - cp
            - /probes/readinessprobe
            - /opt/scripts/readinessprobe
          image: examplesaas.azurecr.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.23
          imagePullPolicy: Always
          name: mongodb-agent-readinessprobe
          resources:
            limits:
              cpu: "1"
              memory: 500M
            requests:
              cpu: 500m
              memory: 400M
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /opt/scripts
              name: agent-scripts
      nodeSelector:
        node.example.com/agentpool: worker
      priorityClassName: platform-system
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 2000
        runAsNonRoot: true
        runAsUser: 2000
      serviceAccount: mongodb-kubernetes-appdb
      serviceAccountName: mongodb-kubernetes-appdb
      terminationGracePeriodSeconds: 30
      tolerations:
        - effect: PreferNoSchedule
          key: node.example.com/agentpool
          operator: Equal
          value: worker
        - effect: NoSchedule
          key: kubernetes.azure.com/scalesetpriority
          operator: Equal
          value: spot
      topologySpreadConstraints:
        - labelSelector:
            matchLabels:
              app: example-application-mongodb-headless
          matchLabelKeys:
            - controller-revision-hash
          maxSkew: 1
          nodeTaintsPolicy: Honor
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
        - labelSelector:
            matchLabels:
              app: example-application-mongodb-headless
          matchLabelKeys:
            - controller-revision-hash
          maxSkew: 1
          nodeTaintsPolicy: Honor
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
      volumes:
        - emptyDir: {}
          name: agent-scripts
        - name: automation-config
          secret:
            defaultMode: 416
            secretName: example-application-mongodb-config
        - emptyDir: {}
          name: healthstatus
        - emptyDir: {}
          name: hooks
        - emptyDir: {}
          name: example-application-mongodb-keyfile
        - emptyDir: {}
          name: tmp
  updateStrategy:
    type: RollingUpdate
  volumeClaimTemplates:
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        creationTimestamp: null
        name: data-volume
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10G
        storageClassName: custom-retain
        volumeMode: Filesystem
      status:
        phase: Pending
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        creationTimestamp: null
        name: logs-volume
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 2G
        storageClassName: custom-retain
        volumeMode: Filesystem
      status:
        phase: Pending
status:
  availableReplicas: 3
  collisionCount: 0
  currentReplicas: 3
  currentRevision: redacted-6778c449fc
  observedGeneration: 779
  readyReplicas: 3
  replicas: 3
  updateRevision: redacted-6778c449fc
  updatedReplicas: 3

I can see in the operator logs that reconciliation started later due to different reason, but stateful set topologySpreadConstraints was not updated. Operator logs:

1768290174064	2026-01-13T07:42:54.064Z	{"level":"info","ts":1768290174.064393,"caller":"agent/agent_readiness.go:62","msg":"All 3 Agents have reached Goal state","ReplicaSet":"example/example-application-mongodb"}
1768290174054	2026-01-13T07:42:54.054Z	{"level":"info","ts":1768290174.0547287,"caller":"controllers/replica_set_controller.go:364","msg":"Creating/Updating AutomationConfig","ReplicaSet":"example/example-application-mongodb"}
1768290174054	2026-01-13T07:42:54.054Z	{"level":"info","ts":1768290174.0545614,"caller":"controllers/replica_set_controller.go:472","msg":"Create/Update operation succeeded","ReplicaSet":"example/example-application-mongodb","operation":"updated"}
1768290174035	2026-01-13T07:42:54.035Z	{"level":"info","ts":1768290174.0352888,"caller":"controllers/replica_set_controller.go:156","msg":"Reconciling MongoDB","ReplicaSet":"example/example-application-mongodb"}
1768290174021	2026-01-13T07:42:54.021Z	{"level":"info","ts":1768290174.021209,"caller":"controllers/mongodb_status_options.go:110","msg":"ReplicaSet is not yet ready, retrying in 10 seconds"}
1768290173983	2026-01-13T07:42:53.983Z	{"level":"info","ts":1768290173.983354,"caller":"controllers/replica_set_controller.go:344","msg":"Creating/Updating StatefulSet for Arbiters","ReplicaSet":"example/example-application-mongodb"}
1768290173874	2026-01-13T07:42:53.874Z	{"level":"info","ts":1768290173.874106,"caller":"controllers/replica_set_controller.go:339","msg":"Creating/Updating StatefulSet","ReplicaSet":"example/example-application-mongodb"}
1768290173874	2026-01-13T07:42:53.874Z	{"level":"info","ts":1768290173.8740652,"caller":"agent/agent_readiness.go:62","msg":"All 3 Agents have reached Goal state","ReplicaSet":"example/example-application-mongodb"}
1768290173853	2026-01-13T07:42:53.853Z	{"level":"info","ts":1768290173.8538957,"caller":"controllers/replica_set_controller.go:364","msg":"Creating/Updating AutomationConfig","ReplicaSet":"example/example-application-mongodb"}
1768290173852	2026-01-13T07:42:53.852Z	{"level":"info","ts":1768290173.852983,"caller":"controllers/replica_set_controller.go:472","msg":"Create/Update operation succeeded","ReplicaSet":"example/example-application-mongodb","operation":"updated"}
1768290173839	2026-01-13T07:42:53.839Z	{"level":"info","ts":1768290173.8394437,"caller":"controllers/replica_set_controller.go:156","msg":"Reconciling MongoDB","ReplicaSet":"example/example-application-mongodb"}

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions