Skip to content

Redis shard expansion failed #2247

New issue
New issue
Open
Open
Redis shard expansion failed#2247
@seek0608

Description

@seek0608
seek0608
opened on Nov 17, 2025

My addon version is 1.0.1, and I am using Redis with a cluster architecture. After expanding the shards, an exception occurred.

apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: redis-sharding3
spec:
  terminationPolicy: Delete
  shardings:
  - name: shard
    shards: 3
    template:
      name: redis
      componentDef: redis-cluster-7
      disableExporter: true
      replicas: 2
      resources:
        limits:
          cpu: '0.1'
          memory: 0.1Gi
        requests:
          cpu: '0.1'
          memory: 0.1Gi
      serviceVersion: 7.2.10
      services:
      - name: redis-advertised
        podService: true
        serviceType: NodePort
      volumeClaimTemplates:
      - name: data
        spec:
          accessModes:
          - ReadWriteOnce
          resources:
            requests:
              storage: 20Gi



apiVersion: operations.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  name: redis-scale-out-fjh4
spec:
  clusterName: redis-sharding3
  type: HorizontalScaling
  horizontalScaling:
    - componentName: shard
      shards: 5

I used the command kubectl logs -f -n kb-system kubeblocks-54f6c8f9bb-lgbgb to check the kubeblocks logs, as detailed below.

2025-11-17T09:30:36.001Z        ERROR   Reconciler error        {"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"redis-sharding3-shard-4wb","namespace":"fujiahao"}, "namespace": "fujiahao", "name": "redis-sharding3-shard-4wb", "reconcileID": "77852a84-b1ed-414f-a912-a1c17f3851a3", "error": "action: postProvision, error: exit code: 1: failed: action failed", "errorVerbose": "action failed\naction: postProvision, error: exit code: 1: failed\ngithub.com/apecloud/kubeblocks/pkg/controller/lifecycle.(*kbagent).formatError.func1\n\t/src/pkg/controller/lifecycle/kbagent.go:345\ngithub.com/apecloud/kubeblocks/pkg/controller/lifecycle.(*kbagent).formatError\n\t/src/pkg/controller/lifecycle/kbagent.go:366\ngithub.com/apecloud/kubeblocks/pkg/controller/lifecycle.(*kbagent).callActionWithSelector\n\t/src/pkg/controller/lifecycle/kbagent.go:316\ngithub.com/apecloud/kubeblocks/pkg/controller/lifecycle.(*kbagent).callAction\n\t/src/pkg/controller/lifecycle/kbagent.go:224\ngithub.com/apecloud/kubeblocks/pkg/controller/lifecycle.(*kbagent).checkedCallAction\n\t/src/pkg/controller/lifecycle/kbagent.go:151\ngithub.com/apecloud/kubeblocks/pkg/controller/lifecycle.(*kbagent).PostProvision\n\t/src/pkg/controller/lifecycle/kbagent.go:65\ngithub.com/apecloud/kubeblocks/controllers/apps/component.(*componentPostProvisionTransformer).postProvision\n\t/src/controllers/apps/component/transformer_component_post_provision.go:90\ngithub.com/apecloud/kubeblocks/controllers/apps/component.(*componentPostProvisionTransformer).Transform\n\t/src/controllers/apps/component/transformer_component_post_provision.go:56\ngithub.com/apecloud/kubeblocks/pkg/controller/graph.TransformerChain.ApplyTo\n\t/src/pkg/controller/graph/transformer.go:57\ngithub.com/apecloud/kubeblocks/controllers/apps/component.(*componentPlanBuilder).Build\n\t/src/controllers/apps/component/component_plan_builder.go:111\ngithub.com/apecloud/kubeblocks/controllers/apps/component.(*ComponentReconciler).Reconcile\n\t/src/controllers/apps/component/component_controller.go:178\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
2025-11-17T09:30:53.385Z        INFO    build error: action: postProvision, error: timedOut: action timed-out   {"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"redis-sharding3-shard-6vn","namespace":"fujiahao"}, "namespace": "fujiahao", "name": "redis-sharding3-shard-6vn", "reconcileID": "68f70d7e-378f-46b4-8148-92586500d1a1", "component": {"name":"redis-sharding3-shard-6vn","namespace":"fujiahao"}}
2025-11-17T09:30:53.393Z        INFO    reconcile object *v1.Component with action STATUS OK    {"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"redis-sharding3-shard-6vn","namespace":"fujiahao"}, "namespace": "fujiahao", "name": "redis-sharding3-shard-6vn", "reconcileID": "68f70d7e-378f-46b4-8148-92586500d1a1", "component": {"name":"redis-sharding3-shard-6vn","namespace":"fujiahao"}}
2025-11-17T09:30:53.393Z        INFO    action: postProvision, error: timedOut: action timed-out        {"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"redis-sharding3-shard-6vn","namespace":"fujiahao"}, "namespace": "fujiahao", "name": "redis-sharding3-shard-6vn", "reconcileID": "68f70d7e-378f-46b4-8148-92586500d1a1", "component": {"name":"redis-sharding3-shard-6vn","namespace":"fujiahao"}}
2025-11-17T09:30:53.393Z        ERROR   Reconciler error        {"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"redis-sharding3-shard-6vn","namespace":"fujiahao"}, "namespace": "fujiahao", "name": "redis-sharding3-shard-6vn", "reconcileID": "68f70d7e-378f-46b4-8148-92586500d1a1", "error": "action: postProvision, error: timedOut: action timed-out", "errorVerbose": "action timed-out\naction: postProvision, error: timedOut\ngithub.com/apecloud/kubeblocks/pkg/controller/lifecycle.(*kbagent).formatError.func1\n\t/src/pkg/controller/lifecycle/kbagent.go:345\ngithub.com/apecloud/kubeblocks/pkg/controller/lifecycle.(*kbagent).formatError\n\t/src/pkg/controller/lifecycle/kbagent.go:364\ngithub.com/apecloud/kubeblocks/pkg/controller/lifecycle.(*kbagent).callActionWithSelector\n\t/src/pkg/controller/lifecycle/kbagent.go:316\ngithub.com/apecloud/kubeblocks/pkg/controller/lifecycle.(*kbagent).callAction\n\t/src/pkg/controller/lifecycle/kbagent.go:224\ngithub.com/apecloud/kubeblocks/pkg/controller/lifecycle.(*kbagent).checkedCallAction\n\t/src/pkg/controller/lifecycle/kbagent.go:151\ngithub.com/apecloud/kubeblocks/pkg/controller/lifecycle.(*kbagent).PostProvision\n\t/src/pkg/controller/lifecycle/kbagent.go:65\ngithub.com/apecloud/kubeblocks/controllers/apps/component.(*componentPostProvisionTransformer).postProvision\n\t/src/controllers/apps/component/transformer_component_post_provision.go:90\ngithub.com/apecloud/kubeblocks/controllers/apps/component.(*componentPostProvisionTransformer).Transform\n\t/src/controllers/apps/component/transformer_component_post_provision.go:56\ngithub.com/apecloud/kubeblocks/pkg/controller/graph.TransformerChain.ApplyTo\n\t/src/pkg/controller/graph/transformer.go:57\ngithub.com/apecloud/kubeblocks/controllers/apps/component.(*componentPlanBuilder).Build\n\t/src/controllers/apps/component/component_plan_builder.go:111\ngithub.com/apecloud/kubeblocks/controllers/apps/component.(*ComponentReconciler).Reconcile\n\t/src/controllers/apps/component/component_controller.go:178\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227


Topology:
COMPONENT          SERVICE-VERSION   INSTANCE                      ROLE        STATUS    AZ       NODE                        CREATED-TIME                 
shard(shard-4wb)   7.2.10            redis-sharding3-shard-4wb-0   primary     Running   160002   10.70.20.119/10.70.20.119   Nov 17,2025 17:29 UTC+0800   
shard(shard-4wb)   7.2.10            redis-sharding3-shard-4wb-1   primary     Running   160002   10.70.19.2/10.70.19.2       Nov 17,2025 17:29 UTC+0800   
shard(shard-6vn)   7.2.10            redis-sharding3-shard-6vn-0   primary     Running   160002   10.70.20.119/10.70.20.119   Nov 17,2025 17:29 UTC+0800   
shard(shard-6vn)   7.2.10            redis-sharding3-shard-6vn-1   secondary   Running   160002   10.70.20.119/10.70.20.119   Nov 17,2025 17:29 UTC+0800   
shard(shard-c2h)   7.2.10            redis-sharding3-shard-c2h-0   primary     Running   160001   10.70.10.30/10.70.10.30     Nov 17,2025 16:50 UTC+0800   
shard(shard-c2h)   7.2.10            redis-sharding3-shard-c2h-1   secondary   Running   160001   10.70.12.141/10.70.12.141   Nov 17,2025 16:51 UTC+0800   
shard(shard-gc5)   7.2.10            redis-sharding3-shard-gc5-0   primary     Running   160001   10.70.12.141/10.70.12.141   Nov 17,2025 16:50 UTC+0800   
shard(shard-gc5)   7.2.10            redis-sharding3-shard-gc5-1   secondary   Running   160001   10.70.12.141/10.70.12.141   Nov 17,2025 16:51 UTC+0800   
shard(shard-h5m)   7.2.10            redis-sharding3-shard-h5m-0   primary     Running   160001   10.70.12.141/10.70.12.141   Nov 17,2025 16:50 UTC+0800   
shard(shard-h5m)   7.2.10            redis-sharding3-shard-h5m-1   secondary   Running   160001   10.70.10.30/10.70.10.30     Nov 17,2025 16:51 UTC+0800

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions