Skip to content

[BUG] mysql orchestrator cluster RebuildInstance Offline Instances timeout #9757

@JashBook

Description

@JashBook

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

  1. creste cluster
apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: orcm-lbypdh
  namespace: default
spec:
  clusterDef: orchestrator
  topology: raft
  terminationPolicy: WipeOut
  services:
    - name: orchestrator
      componentSelector: orchestrator
      spec:
        ports:
          - name: orc-http
            port: 80
  componentSpecs:
    - name: orchestrator
      serviceVersion: 3.2.6
      replicas: 3
      resources:
        requests:
          cpu: 500m
          memory: 0.5Gi
        limits:
          cpu: 500m
          memory: 0.5Gi
      volumeClaimTemplates:
        - name: data
          spec:
            storageClassName: 
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi

mysql cluster

apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: mysql-lbypdh
  namespace: default
spec:
  clusterDef: mysql
  topology: orc
  terminationPolicy: WipeOut
  componentSpecs:
    - name: mysql
      serviceVersion: 8.0.30
      
      disableExporter: true
      replicas: 2
      resources:
        limits:
          cpu: 500m
          memory: 0.5Gi
        requests:
          cpu: 500m
          memory: 0.5Gi
      volumeClaimTemplates:
        - name: data
          spec:
            storageClassName: 
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
      serviceRefs:
      - name: orchestrator
        namespace: default
        clusterServiceSelector:
          cluster: orcm-lbypdh
          service:
            component: orchestrator
            service: orchestrator
            port: orc-http
          credential:
            component: orchestrator
            name: orchestrator
  1. insert data
kubectl get secrets mysql-lbypdh-mysql-account-root -o jsonpath="{.data.password}"|base64 -d
9D9t0d2VP3
apiVersion: v1
kind: Pod
metadata:
  name: test-db-client-executionloop-mysql-lbypdh
  namespace: default
spec:
  containers:
    - name: test-dbclient
      imagePullPolicy: IfNotPresent
      image: apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/dbclient:test
      args:
        - "--host"
        - "mysql-lbypdh-mysql-server.default.svc.cluster.local"
        - "--user"
        - "root"
        - "--password"
        - "9D9t0d2VP3"
        - "--port"
        - "3306"
        - "--dbtype"
        - "mysql"
        - "--test"
        - "executionloop"
        - "--duration"
        - "60"
        - "--interval"
        - "1"
  restartPolicy: Never
echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-lbypdh-mysql-server.default.svc.cluster.local -uroot -p'9D9t0d2VP3' " | kubectl exec -it mysql-lbypdh-mysql-0 --namespace default -- bash
Defaulted container "mysql" out of: mysql, kbagent, config-manager, init-data (init), init-jq (init), init-kbagent (init), kbagent-worker (init)
Unable to use a TTY - input is not a terminal or the right kind of file
mysql: [Warning] Using a password on the command line interface can be insecure.
count(*)
17311
  1. backup
kbcli cluster backup mysql-lbypdh --method xtrabackup 
Backup backup-default-mysql-lbypdh-20250918181756 created successfully, you can view the progress:
	kbcli cluster list-backups --names=backup-default-mysql-lbypdh-20250918181756 -n default

kubectl get backup backup-default-mysql-lbypdh-20250918181756
NAME                                         POLICY                             METHOD       REPO                    STATUS      TOTAL-SIZE   DURATION   DELETION-POLICY   CREATION-TIME          COMPLETION-TIME        EXPIRATION-TIME
backup-default-mysql-lbypdh-20250918181756   mysql-lbypdh-mysql-backup-policy   xtrabackup   backuprepo-kbcli-test   Completed   12015851     10s        Delete            2025-09-18T10:17:56Z   2025-09-18T10:18:06Z  
kbcli cluster list-instances abk-lbypdh --namespace default
    
NAME                 NAMESPACE   CLUSTER      COMPONENT   STATUS    ROLE        ACCESSMODE   AZ               CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE     NODE                          CREATED-TIME                 
abk-lbypdh-mysql-0   default     abk-lbypdh   mysql       Running   primary                  cn-guangzhou-b   500m / 500m          512Mi / 512Mi           data:20Gi   192.168.0.135/192.168.0.135   Sep 18,2025 18:18 UTC+0800   
abk-lbypdh-mysql-1   default     abk-lbypdh   mysql       Running   secondary                cn-guangzhou-b   500m / 500m          512Mi / 512Mi           data:20Gi   192.168.0.54/192.168.0.54     Sep 18,2025 18:19 UTC+0800   
kubectl create -f -<<EOF
apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  generateName: mysql-lbypdh-rebuildinstance-
  namespace: default
spec:
  type: RebuildInstance
  clusterName: mysql-lbypdh
  force: true
  rebuildFrom:
    - componentName: mysql
      instances:
        - name: mysql-lbypdh-mysql-1
      backupName: backup-default-mysql-lbypdh-20250918181756
      inPlace: false
      restoreEnv:
        - name: BACKUP_FOR_STANDBY
          value: "true"
EOF
  1. See error
kubectl get cluster mysql-lbypdh
NAME           CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS     AGE
mysql-lbypdh   mysql                WipeOut              Updating   15h
➜  ~ 
➜  ~ kubectl get pod -l app.kubernetes.io/instance=mysql-lbypdh                           

NAME                   READY   STATUS    RESTARTS   AGE
mysql-lbypdh-mysql-0   3/3     Running   0          15h
mysql-lbypdh-mysql-1   3/3     Running   0          15h
mysql-lbypdh-mysql-2   3/3     Running   0          15h
➜  ~ 
➜  ~ kubectl get cmp -l app.kubernetes.io/instance=mysql-lbypdh

NAME                 DEFINITION            SERVICE-VERSION   STATUS     AGE
mysql-lbypdh-mysql   mysql-orc-8.0-1.0.1   8.0.30            Updating   15h

kubectl get ops -l app.kubernetes.io/instance=mysql-lbypdh

NAME                                 TYPE              CLUSTER        STATUS    PROGRESS   AGE
mysql-lbypdh-rebuildinstance-45cdw   RebuildInstance   mysql-lbypdh   Running   0/1        15h

describe

kubectl describe ops mysql-lbypdh-rebuildinstance-45cdw
Name:         mysql-lbypdh-rebuildinstance-45cdw
Namespace:    default
Labels:       app.kubernetes.io/instance=mysql-lbypdh
              operations.kubeblocks.io/ops-type=RebuildInstance
Annotations:  <none>
API Version:  operations.kubeblocks.io/v1alpha1
Kind:         OpsRequest
Metadata:
  Creation Timestamp:  2025-09-18T10:20:28Z
  Finalizers:
    opsrequest.kubeblocks.io/finalizer
  Generate Name:  mysql-lbypdh-rebuildinstance-
  Generation:     1
  Owner References:
    API Version:     apps.kubeblocks.io/v1
    Kind:            Cluster
    Name:            mysql-lbypdh
    UID:             37a4aef0-2e8c-4eb7-986b-e03f5e3de3d5
  Resource Version:  98882343
  UID:               effabdc9-359d-4fdf-8e13-c32f62ae2eaf
Spec:
  Cluster Name:                    mysql-lbypdh
  Enqueue On Force:                false
  Force:                           true
  Pre Condition Deadline Seconds:  0
  Rebuild From:
    Backup Name:     backup-default-mysql-lbypdh-20250918181756
    Component Name:  mysql
    In Place:        false
    Instances:
      Name:  mysql-lbypdh-mysql-1
    Restore Env:
      Name:   BACKUP_FOR_STANDBY
      Value:  true
  Type:       RebuildInstance
Status:
  Cluster Generation:  2
  Components:
    Mysql:
      Progress Details:
        Message:     Scaling out a new pod: mysql-lbypdh-mysql-2, status: Available
        Object Key:  Pod/mysql-lbypdh-mysql-1
        Start Time:  2025-09-18T10:20:28Z
        Status:      Processing
  Conditions:
    Last Transition Time:  2025-09-18T10:20:28Z
    Message:               wait for the controller to process the OpsRequest: mysql-lbypdh-rebuildinstance-45cdw in Cluster: mysql-lbypdh
    Reason:                WaitForProgressing
    Status:                True
    Type:                  WaitForProgressing
    Last Transition Time:  2025-09-18T10:20:28Z
    Message:               OpsRequest: mysql-lbypdh-rebuildinstance-45cdw is validated
    Reason:                ValidateOpsRequestPassed
    Status:                True
    Type:                  Validated
    Last Transition Time:  2025-09-18T10:20:28Z
    Message:               Start to rebuild the instances in Cluster: mysql-lbypdh
    Reason:                StartToRebuildInstances
    Status:                True
    Type:                  InstancesRebuilding
  Last Configuration:
    Components:
      Mysql:
        Replicas:   2
  Phase:            Running
  Progress:         0/1
  Start Timestamp:  2025-09-18T10:20:28Z
Events:             <none>
kubectl describe cmp mysql-lbypdh-mysql 
Name:         mysql-lbypdh-mysql
Namespace:    default
Labels:       app.kubernetes.io/component=mysql-orc-8.0-1.0.1
              app.kubernetes.io/instance=mysql-lbypdh
              app.kubernetes.io/managed-by=kubeblocks
              apps.kubeblocks.io/component-name=mysql
              componentdefinition.kubeblocks.io/name=mysql-orc-8.0-1.0.1
Annotations:  apps.kubeblocks.io/cluster-uid: 37a4aef0-2e8c-4eb7-986b-e03f5e3de3d5
              kubeblocks.io/crd-api-version: apps.kubeblocks.io/v1
              kubeblocks.io/generation: 4
              kubeblocks.io/post-provision-done: 2025-09-18T10:16:06.159264895Z
API Version:  apps.kubeblocks.io/v1
Kind:         Component
Metadata:
  Creation Timestamp:  2025-09-18T10:14:55Z
  Finalizers:
    cluster.kubeblocks.io/finalizer
    component.kubeblocks.io/finalizer
  Generation:  4
  Owner References:
    API Version:           apps.kubeblocks.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Cluster
    Name:                  mysql-lbypdh
    UID:                   37a4aef0-2e8c-4eb7-986b-e03f5e3de3d5
  Resource Version:        98882552
  UID:                     65e67bf6-5285-4637-b4ee-c1212c810574
Spec:
  Comp Def:  mysql-orc-8.0-1.0.1
  Configs:
    Config Map:
      Name:          mysql-lbypdh-mysql-mysql-replication-config
    Name:            mysql-replication-config
  Disable Exporter:  true
  Offline Instances:
    mysql-lbypdh-mysql-1
  Pod Update Policy:  PreferInPlace
  Replicas:           2
  Resources:
    Limits:
      Cpu:     500m
      Memory:  512Mi
    Requests:
      Cpu:     500m
      Memory:  512Mi
  Service Refs:
    Cluster Service Selector:
      Cluster:  orcm-lbypdh
      Credential:
        Component:  orchestrator
        Name:       orchestrator
      Service:
        Component:     orchestrator
        Port:          orc-http
        Service:       orchestrator
    Name:              orchestrator
    Namespace:         default
  Service Version:     8.0.30
  Termination Policy:  WipeOut
  Volume Claim Templates:
    Name:  data
    Spec:
      Access Modes:
        ReadWriteOnce
      Resources:
        Requests:
          Storage:  20Gi
Status:
  Conditions:
    Last Transition Time:  2025-09-18T10:14:55Z
    Message:               The operator has started the provisioning of Cluster: mysql-lbypdh-mysql
    Observed Generation:   4
    Reason:                PreCheckSucceed
    Status:                True
    Type:                  ProvisioningStarted
    Last Transition Time:  2025-09-18T10:20:28Z
    Message:               the component phase is Updating
    Observed Generation:   3
    Reason:                Unavailable
    Status:                False
    Type:                  Available
    Last Transition Time:  2025-09-18T10:16:06Z
    Message:               proxysql:,root:
    Observed Generation:   3
    Reason:                AllProvisioned
    Status:                True
    Type:                  SystemAccountProvision
  Observed Generation:     3
  Phase:                   Updating
Events:                    <none>

logs pod 1 kbagent

kubectl logs mysql-lbypdh-mysql-1 kbagent --tail 10 
2025-09-19T02:07:57Z	INFO	Action Executed	{"action": "switchover", "result": "switchover not triggered for primary, nothing to do, exit 0.\n"}
2025-09-19T02:07:57Z	INFO	HTTP API Called	{"user-agent": "Go-http-client/1.1", "method": "POST", "path": "/v1.0/action", "status code": 200, "cost": 1}
2025-09-19T02:08:28Z	INFO	Action Executed	{"action": "memberLeave", "result": "2025-09-19 02:07:57 ERROR Unable to determine cluster name. clusterHint=mysql-lbypdh-mysql-1\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\n: timedOut"}
2025-09-19T02:08:28Z	INFO	HTTP API Called	{"user-agent": "Go-http-client/1.1", "method": "POST", "path": "/v1.0/action", "status code": 200, "cost": 30925}
2025-09-19T02:08:28Z	INFO	Action Executed	{"action": "switchover", "result": "switchover not triggered for primary, nothing to do, exit 0.\n"}
2025-09-19T02:08:28Z	INFO	HTTP API Called	{"user-agent": "Go-http-client/1.1", "method": "POST", "path": "/v1.0/action", "status code": 200, "cost": 1}
2025-09-19T02:08:59Z	INFO	Action Executed	{"action": "memberLeave", "result": "2025-09-19 02:08:28 ERROR Unable to determine cluster name. clusterHint=mysql-lbypdh-mysql-1\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\n: timedOut"}
2025-09-19T02:08:59Z	INFO	HTTP API Called	{"user-agent": "Go-http-client/1.1", "method": "POST", "path": "/v1.0/action", "status code": 200, "cost": 30924}
2025-09-19T02:08:59Z	INFO	Action Executed	{"action": "switchover", "result": "switchover not triggered for primary, nothing to do, exit 0.\n"}
2025-09-19T02:08:59Z	INFO	HTTP API Called	{"user-agent": "Go-http-client/1.1", "method": "POST", "path": "/v1.0/action", "status code": 200, "cost": 1}

logs KubeBlocks pod

kubectl logs -n kb-system kubeblocks-6d88d97b-qknp9 --tail 150|grep mysql-lbypdh-mysql 
Defaulted container "manager" out of: manager, tools (init)
2025-09-19T02:10:32.016Z	ERROR	leave member at scale-in error	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"mysql-lbypdh-mysql","namespace":"default"}, "namespace": "default", "name": "mysql-lbypdh-mysql", "reconcileID": "3efc8347-e43a-43f5-9b51-d495d437468a", "component": {"name":"mysql-lbypdh-mysql","namespace":"default"}, "error": "requeue after: 1s as: [action: memberLeave, error: 2025-09-19 02:10:01 ERROR Unable to determine cluster name. clusterHint=mysql-lbypdh-mysql-1\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance:mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\n: timedOut: action timed-out]"}
2025-09-19T02:10:32.016Z	ERROR	leave member at scale-in error	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"mysql-lbypdh-mysql","namespace":"default"}, "namespace": "default", "name": "mysql-lbypdh-mysql", "reconcileID": "3efc8347-e43a-43f5-9b51-d495d437468a", "component": {"name":"mysql-lbypdh-mysql","namespace":"default"}, "error": "requeue after: 1s as: [action: memberLeave, error: 2025-09-19 02:10:01 ERROR Unable to determine cluster name. clusterHint=mysql-lbypdh-mysql-1\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysql-1:3306\nCannot read instance: mysql-lbypdh-mysq-1:3306\n: timedOut: action timed-out]"}
2025-09-19T02:10:32.017Z	INFO	build error: requeue after: 1s as: [action: memberLeave, error: 2025-09-19 02:10:01 ERROR Unable to determine cluster name. clusterHint=mysql-lbypdh-mysql-1
Cannot read instance: mysql-lbypdh-mysql-1:3306
Cannot read instance: mysql-lbypdh-mysql-1:3306
Cannot read instance: mysql-lbypdh-mysql-1:3306
Cannot read instance: mysql-lbypdh-mysql-1:3306
Cannot read instance: mysql-lbypdh-mysql-1:3306
Cannot read instance: mysql-lbypdh-mysql-1:3306
Cannot read instance: mysql-lbypdh-mysql-1:3306
Cannot read instance: mysql-lbypdh-mysql-1:3306
Cannot read instance: mysql-lbypdh-mysql-1:3306
Cannot read instance: mysql-lbypdh-mysql-1:3306
Cannot read instance: mysql-lbypdh-mysql-1:3306
Cannot read instance: mysql-lbypdh-mysql-1:3306
Cannot read instance: mysql-lbypdh-mysql-1:3306
Cannot read instance: mysql-lbypdh-mysql-1:3306
Cannot read instance: mysql-lbypdh-mysql-1:3306
Cannot read instance: mysql-lbypdh-mysql-1:3306
Cannot read instance: mysql-lbypdh-mysql-1:3306
Cannot read instance: mysql-lbypdh-mysql-1:3306
Cannot read instance: mysql-lbypdh-mysql-1:3306
Cannot read instance: mysql-lbypdh-mysql-1:3306
Cannot read instance: mysql-lbypdh-mysql-1:3306
Cannot read instance: mysql-lbypdh-mysql-1:3306
Cannot read instance: mysql-lbypdh-mysql-1:3306
Cannot read instance: mysql-lbypdh-mysql-1:3306
Cannot read instance: mysql-lbypdh-mysql-1:3306
Cannot read instance: mysql-lbypdh-mysql-1:3306
Cannot read instance: mysql-lbypdh-mysql-1:3306
Cannot read instance: mysql-lbypdh-mysql-1:3306
Cannot read instance: mysql-lbypdh-mysql-1:3306
: timedOut: action timed-out]	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"mysql-lbypdh-mysql","namespace":"default"}, "namespace": "default", "name": "mysql-lbypdh-mysql", "reconcileID": "3efc8347-e43a-43f5-9b51-d495d437468a", "component": {"name":"mysql-lbypdh-mysql","namespace":"default"}}
2025-09-19T02:10:32.023Z	INFO	reconcile object *v1.Component with action STATUS OK	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"mysql-lbypdh-mysql","namespace":"default"}, "namespace": "default", "name": "mysql-lbypdh-mysql", "reconcileID": "3efc8347-e43a-43f5-9b51-d495d437468a", "component": {"name":"mysql-lbypdh-mysql","namespace":"default"}}
2025-09-19T02:10:32.083Z	INFO	leave member at scaling-in	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"mysql-lbypdh-mysql","namespace":"default"}, "namespace": "default", "name": "mysql-lbypdh-mysql", "reconcileID": "fc5c7716-d128-4d1a-83d2-3b396c96e58f", "component": {"name":"mysql-lbypdh-mysql","namespace":"default"}, "delete replicas": ["mysql-lbypdh-mysql-1"], "joined replicas": ["mysql-lbypdh-mysql-1"], "has member-leave action defined": true}
2025-09-19T02:10:32.085Z	INFO	successfully call switchover action for pod	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"mysql-lbypdh-mysql","namespace":"default"}, "namespace": "default", "name": "mysql-lbypdh-mysql", "reconcileID": "fc5c7716-d128-4d1a-83d2-3b396c96e58f", "component": {"name":"mysql-lbypdh-mysql","namespace":"default"}, "pod": "mysql-lbypdh-mysql-1"}

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

Metadata

Metadata

Assignees

Labels

kind/bugSomething isn't working

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions