-
Notifications
You must be signed in to change notification settings - Fork 237
Open
Labels
kind/bugSomething isn't workingSomething isn't working
Description
Describe the bug
Kubernetes: v1.33.3+k3s1
KubeBlocks: 1.0.1
kbcli: 1.0.1
when start cluster, cluster always stuck in Updating state.
pod master-0 always in PodInitializing, and did not see error log.
lupin59-data-0 3/3 Running 0 26m
lupin59-master-0 0/3 PodInitializing 0 26m
To Reproduce
- put below yaml file in helm and install it
{{- $ca := genCA "KubeBlocks" 36500 }}
{{- $cert := genSignedCert "elasticsearch" (list "127.0.0.1" "::1") (list "localhost" "*.cluster.local") 36500 $ca }}
apiVersion: v1
kind: Secret
metadata:
name: es-tls-secret
namespace: default
labels:
helm.sh/chart: elasticsearch-cluster-1.0.1
app.kubernetes.io/name: elasticsearch-cluster
app.kubernetes.io/instance: elasticsearch
app.kubernetes.io/version: "8.8.2"
type: Opaque
stringData:
ca.crt: {{ $ca.Cert | quote }}
tls.crt: {{ $cert.Cert | quote }}
tls.key: {{ $cert.Key | quote }}
---
apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
labels:
app.kubernetes.io/instance: lupin59
app.kubernetes.io/version: 8.8.2
helm.sh/chart: elasticsearch-cluster-1.0.0
name: lupin59
namespace: default
spec:
componentSpecs:
- componentDef: elasticsearch-8
configs:
- name: es-cm
variables:
roles: data,ingest,transform
version: 8.15.5
disableExporter: true
name: data
replicas: 1
tls: true
issuer:
name: UserProvided
secretRef:
namespace: default
name: es-tls-secret
ca: ca.crt
cert: tls.crt
key: tls.key
resources:
limits:
cpu: "1"
memory: 2Gi
requests:
cpu: "1"
memory: 2Gi
schedulingPolicy:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/instance: lupin59
apps.kubeblocks.io/component-name: mdit
topologyKey: kubernetes.io/hostname
weight: 100
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app.kubernetes.io/instance: lupin59
apps.kubeblocks.io/component-name: mdit
topologyKey: kubernetes.io/hostname
serviceVersion: 8.15.5
volumeClaimTemplates:
- name: data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
- componentDef: elasticsearch-8
configs:
- name: es-cm
variables:
roles: master
version: 8.15.5
disableExporter: true
name: master
replicas: 1
tls: true
issuer:
name: UserProvided
secretRef:
namespace: default
name: es-tls-secret
ca: ca.crt
cert: tls.crt
key: tls.key
resources:
limits:
cpu: "1"
memory: 2Gi
requests:
cpu: "1"
memory: 2Gi
schedulingPolicy:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/instance: lupin59
apps.kubeblocks.io/component-name: mdit
topologyKey: kubernetes.io/hostname
weight: 100
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app.kubernetes.io/instance: lupin59
apps.kubeblocks.io/component-name: mdit
topologyKey: kubernetes.io/hostname
serviceVersion: 8.15.5
volumeClaimTemplates:
- name: data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
terminationPolicy: Delete
- stop cluster and then start it
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
- OS: [e.g. iOS]
- Browser [e.g. chrome, safari]
- Version [e.g. 22]
Additional context
master pod describe
Name: lupin59-master-0
Namespace: default
Priority: 0
Service Account: kb-elasticsearch-8-1.0.1
Node: olares/192.168.31.197
Start Time: Sun, 28 Sep 2025 15:06:08 +0800
Labels: app.kubernetes.io/component=elasticsearch-8-1.0.1
app.kubernetes.io/instance=lupin59
app.kubernetes.io/managed-by=kubeblocks
apps.kubeblocks.io/component-name=master
apps.kubeblocks.io/pod-name=lupin59-master-0
apps.kubeblocks.io/release-phase=stable
apps.kubeblocks.io/service-version=8.15.5
controller-revision-hash=5f5b454657
workloads.kubeblocks.io/instance=lupin59-master
workloads.kubeblocks.io/managed-by=InstanceSet
Annotations: cni.projectcalico.org/containerID: adb5ff6b3e741bb65b61ba53093ed2563db3525a6317a4d844ddafc433c4905d
cni.projectcalico.org/podIP: 10.233.98.217/32
cni.projectcalico.org/podIPs: 10.233.98.217/32
Status: Pending
IP: 10.233.98.217
IPs:
IP: 10.233.98.217
Controlled By: InstanceSet/lupin59-master
Init Containers:
prepare-plugins:
Container ID: containerd://41a2f5aca3f9e4319efc90c9b3863547ce63f690d5af22962167a622b188941f
Image: docker.io/library/elasticsearch:8.15.5
Image ID: docker.io/library/elasticsearch@sha256:82d39a05c027480938d5d405d7ddd933f221effcd1d8a33356f0dd0212640df9
Port: <none>
Host Port: <none>
Command:
sh
-c
if [ -d /plugins ]; then
echo "install plugins: $(ls /plugins)"
cp -r /plugins/* /tmp/plugins/
else
echo "there is no plugins"
fi
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sun, 28 Sep 2025 15:06:09 +0800
Finished: Sun, 28 Sep 2025 15:06:09 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 0
memory: 0
Requests:
cpu: 0
memory: 0
Environment Variables from:
lupin59-master-env ConfigMap Optional: false
Environment:
ELASTIC_PASSWORD: <set to the key 'password' in secret 'lupin59-master-account-elastic'> Optional: false
ELASTIC_USER_PASSWORD: <set to the key 'password' in secret 'lupin59-master-account-elastic'> Optional: false
KIBANA_SYSTEM_USER_PASSWORD: <set to the key 'password' in secret 'lupin59-master-account-kibana-system'> Optional: false
Mounts:
/tmp/plugins from plugins (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-k2twz (ro)
install-plugins:
Container ID: containerd://b77b4bb74fb45b21fadcd3292077f3b75629b3d73d89f6c98ecc13e4ca44d4e3
Image: docker.io/library/elasticsearch:8.15.5
Image ID: docker.io/library/elasticsearch@sha256:82d39a05c027480938d5d405d7ddd933f221effcd1d8a33356f0dd0212640df9
Port: <none>
Host Port: <none>
Command:
sh
-c
set -x
sh /mnt/remote-scripts/install-plugins.sh
sh /mnt/remote-scripts/prepare-fs.sh
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sun, 28 Sep 2025 15:06:10 +0800
Finished: Sun, 28 Sep 2025 15:06:10 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 0
memory: 0
Requests:
cpu: 0
memory: 0
Environment Variables from:
lupin59-master-env ConfigMap Optional: false
Environment:
ELASTIC_PASSWORD: <set to the key 'password' in secret 'lupin59-master-account-elastic'> Optional: false
ELASTIC_USER_PASSWORD: <set to the key 'password' in secret 'lupin59-master-account-elastic'> Optional: false
KIBANA_SYSTEM_USER_PASSWORD: <set to the key 'password' in secret 'lupin59-master-account-kibana-system'> Optional: false
Mounts:
/mnt/local-bin from local-bin (rw)
/mnt/local-config from local-config (rw)
/mnt/local-plugins from local-plugins (rw)
/mnt/remote-config from es-cm (ro)
/mnt/remote-scripts from scripts (ro)
/tmp/plugins from plugins (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-k2twz (ro)
install-es-agent:
Container ID: containerd://b499a27e8e098506925f8ffd7f849a75b5fbd8c6102caa5f36b7d7a73c3952bc
Image: docker.io/apecloud/elasticsearch-agent:0.1.0
Image ID: docker.io/apecloud/elasticsearch-agent@sha256:4cf98d102c7c1ce7d5d570c7b151838a3c060e65b65b68f2a1750a95b5b8dc5d
Port: <none>
Host Port: <none>
Command:
sh
-c
cp /usr/local/bin/agent /mnt/local-bin/es-agent
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sun, 28 Sep 2025 15:06:11 +0800
Finished: Sun, 28 Sep 2025 15:06:11 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 0
memory: 0
Requests:
cpu: 0
memory: 0
Environment Variables from:
lupin59-master-env ConfigMap Optional: false
Environment:
ELASTIC_PASSWORD: <set to the key 'password' in secret 'lupin59-master-account-elastic'> Optional: false
ELASTIC_USER_PASSWORD: <set to the key 'password' in secret 'lupin59-master-account-elastic'> Optional: false
KIBANA_SYSTEM_USER_PASSWORD: <set to the key 'password' in secret 'lupin59-master-account-kibana-system'> Optional: false
Mounts:
/mnt/local-bin from local-bin (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-k2twz (ro)
init-kbagent:
Container ID: containerd://7577e9acdfe7766ec40086cdd48df3dd3ec4008916dab6f633309bf76806cc91
Image: apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/kubeblocks-tools:1.0.1
Image ID: apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/kubeblocks-tools@sha256:6e0084ec006f707226b29e30b9e6e81d3a2d454152e1a6b4bf5dfdc60edf17c8
Port: <none>
Host Port: <none>
Command:
cp
-r
/bin/kbagent
/kubeblocks/
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sun, 28 Sep 2025 15:06:12 +0800
Finished: Sun, 28 Sep 2025 15:06:12 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 0
memory: 0
Requests:
cpu: 0
memory: 0
Environment Variables from:
lupin59-master-env ConfigMap Optional: false
Environment:
ELASTIC_PASSWORD: <set to the key 'password' in secret 'lupin59-master-account-elastic'> Optional: false
ELASTIC_USER_PASSWORD: <set to the key 'password' in secret 'lupin59-master-account-elastic'> Optional: false
KIBANA_SYSTEM_USER_PASSWORD: <set to the key 'password' in secret 'lupin59-master-account-kibana-system'> Optional: false
Mounts:
/kubeblocks from kubeblocks (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-k2twz (ro)
kbagent-worker:
Container ID: containerd://ce53d7b4e39351c837d71e501a273d3c06a5d72b8d2f193b228d3d38bdcf8ca3
Image: docker.io/apecloud/curl-jq:0.1.0
Image ID: docker.io/apecloud/curl-jq@sha256:c13c8973db9bd23b7c3b77654ba18105f3882fb6f655f7b313145154e401a672
Port: <none>
Host Port: <none>
Command:
/kubeblocks/kbagent
Args:
--server=false
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sun, 28 Sep 2025 15:06:13 +0800
Finished: Sun, 28 Sep 2025 15:06:13 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 0
memory: 0
Requests:
cpu: 0
memory: 0
Environment Variables from:
lupin59-master-env ConfigMap Optional: false
Environment:
ELASTIC_PASSWORD: <set to the key 'password' in secret 'lupin59-master-account-elastic'> Optional: false
ELASTIC_USER_PASSWORD: <set to the key 'password' in secret 'lupin59-master-account-elastic'> Optional: false
KIBANA_SYSTEM_USER_PASSWORD: <set to the key 'password' in secret 'lupin59-master-account-kibana-system'> Optional: false
KB_AGENT_NAMESPACE: default (v1:metadata.namespace)
KB_AGENT_POD_NAME: lupin59-master-0 (v1:metadata.name)
KB_AGENT_POD_UID: (v1:metadata.uid)
KB_AGENT_NODE_NAME: (v1:spec.nodeName)
KB_AGENT_ACTION: [{"name":"memberLeave","exec":{"command":["/bin/sh","-c","/mnt/remote-scripts/member-leave.sh"]}}]
KB_AGENT_PROBE: null
Mounts:
/kubeblocks from kubeblocks (rw)
/mnt/remote-config from es-cm (ro)
/mnt/remote-scripts from scripts (ro)
/tmp from tmp-volume (rw)
/usr/share/elasticsearch/bin from local-bin (rw)
/usr/share/elasticsearch/config from local-config (rw)
/usr/share/elasticsearch/data from data (rw)
/usr/share/elasticsearch/logs from log (rw)
/usr/share/elasticsearch/plugins from local-plugins (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-k2twz (ro)
Containers:
elasticsearch:
Container ID:
Image: docker.io/library/elasticsearch:8.15.5
Image ID:
Ports: 9200/TCP, 9300/TCP
Host Ports: 0/TCP, 0/TCP
Command:
sh
-c
cp /etc/pki/tls/* /usr/share/elasticsearch/config/
# remove initial master nodes block if cluster has been formed
if [ -f "${CLUSTER_FORMED_FILE}" ]; then
sed -i '/# INITIAL_MASTER_NODES_BLOCK_START/,/# INITIAL_MASTER_NODES_BLOCK_END/d' config/elasticsearch.yml
fi
if [ -f /bin/tini ]; then
/bin/tini -- /usr/local/bin/docker-entrypoint.sh
else
/tini -- /usr/local/bin/docker-entrypoint.sh
fi
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Limits:
cpu: 1
memory: 2Gi
Requests:
cpu: 1
memory: 2Gi
Readiness: exec [bash -c /mnt/remote-scripts/readiness-probe-script.sh] delay=30s timeout=5s period=5s #success=1 #failure=3
Environment Variables from:
lupin59-master-env ConfigMap Optional: false
Environment:
ELASTIC_PASSWORD: <set to the key 'password' in secret 'lupin59-master-account-elastic'> Optional: false
ELASTIC_USER_PASSWORD: <set to the key 'password' in secret 'lupin59-master-account-elastic'> Optional: false
KIBANA_SYSTEM_USER_PASSWORD: <set to the key 'password' in secret 'lupin59-master-account-kibana-system'> Optional: false
POD_IP: (v1:status.podIP)
POD_NAME: lupin59-master-0 (v1:metadata.name)
NODE_NAME: (v1:spec.nodeName)
KB_NAMESPACE: default (v1:metadata.namespace)
POD_FQDN: $(POD_NAME).$(ES_COMPONENT_NAME)-headless.$(KB_NAMESPACE).svc.$(CLUSTER_DOMAIN)
READINESS_PROBE_PROTOCOL: http
NSS_SDB_USE_CACHE: no
CLUSTER_FORMED_FILE: /usr/share/elasticsearch/data/cluster-formed
Mounts:
/etc/pki/tls from tls (ro)
/mnt/remote-config from es-cm (ro)
/mnt/remote-scripts from scripts (ro)
/tmp from tmp-volume (rw)
/usr/share/elasticsearch/bin from local-bin (rw)
/usr/share/elasticsearch/config from local-config (rw)
/usr/share/elasticsearch/data from data (rw)
/usr/share/elasticsearch/logs from log (rw)
/usr/share/elasticsearch/plugins from local-plugins (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-k2twz (ro)
es-agent:
Container ID:
Image: docker.io/library/elasticsearch:8.15.5
Image ID:
Port: 8080/TCP
Host Port: 0/TCP
Command:
/usr/share/elasticsearch/bin/es-agent
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Limits:
cpu: 0
memory: 0
Requests:
cpu: 0
memory: 0
Liveness: http-get http://:agent/health delay=10s timeout=5s period=10s #success=1 #failure=3
Readiness: http-get http://:agent/health delay=5s timeout=5s period=5s #success=1 #failure=3
Environment Variables from:
lupin59-master-env ConfigMap Optional: false
Environment:
ELASTIC_PASSWORD: <set to the key 'password' in secret 'lupin59-master-account-elastic'> Optional: false
ELASTIC_USER_PASSWORD: <set to the key 'password' in secret 'lupin59-master-account-elastic'> Optional: false
KIBANA_SYSTEM_USER_PASSWORD: <set to the key 'password' in secret 'lupin59-master-account-kibana-system'> Optional: false
POD_IP: (v1:status.podIP)
POD_NAME: lupin59-master-0 (v1:metadata.name)
KB_NAMESPACE: default (v1:metadata.namespace)
POD_FQDN: $(POD_NAME).$(ES_COMPONENT_NAME)-headless.$(KB_NAMESPACE).svc.$(CLUSTER_DOMAIN)
NODE_NAME: (v1:spec.nodeName)
ELASTIC_USERNAME: elastic
ELASTIC_PASSWORD: $(ELASTIC_USER_PASSWORD)
ELASTICSEARCH_HOST: localhost
ELASTICSEARCH_PORT: 9200
AGENT_PORT: 8080
Mounts:
/etc/pki/tls from tls (ro)
/usr/share/elasticsearch/bin from local-bin (rw)
/usr/share/elasticsearch/config from local-config (rw)
/usr/share/elasticsearch/data from data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-k2twz (ro)
kbagent:
Container ID:
Image: docker.io/apecloud/curl-jq:0.1.0
Image ID:
Ports: 3501/TCP, 3502/TCP
Host Ports: 0/TCP, 0/TCP
Command:
/kubeblocks/kbagent
Args:
--port
3501
--streaming-port
3502
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Limits:
cpu: 0
memory: 0
Requests:
cpu: 0
memory: 0
Startup: tcp-socket :3501 delay=0s timeout=1s period=10s #success=1 #failure=3
Environment Variables from:
lupin59-master-env ConfigMap Optional: false
Environment:
ELASTIC_PASSWORD: <set to the key 'password' in secret 'lupin59-master-account-elastic'> Optional: false
ELASTIC_USER_PASSWORD: <set to the key 'password' in secret 'lupin59-master-account-elastic'> Optional: false
KIBANA_SYSTEM_USER_PASSWORD: <set to the key 'password' in secret 'lupin59-master-account-kibana-system'> Optional: false
KB_AGENT_NAMESPACE: default (v1:metadata.namespace)
KB_AGENT_POD_NAME: lupin59-master-0 (v1:metadata.name)
KB_AGENT_POD_UID: (v1:metadata.uid)
KB_AGENT_NODE_NAME: (v1:spec.nodeName)
KB_AGENT_ACTION: [{"name":"memberLeave","exec":{"command":["/bin/sh","-c","/mnt/remote-scripts/member-leave.sh"]}}]
KB_AGENT_PROBE: null
Mounts:
/etc/pki/tls from tls (ro)
/kubeblocks from kubeblocks (rw)
/mnt/remote-config from es-cm (ro)
/mnt/remote-scripts from scripts (ro)
/tmp from tmp-volume (rw)
/usr/share/elasticsearch/bin from local-bin (rw)
/usr/share/elasticsearch/config from local-config (rw)
/usr/share/elasticsearch/data from data (rw)
/usr/share/elasticsearch/logs from log (rw)
/usr/share/elasticsearch/plugins from local-plugins (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-k2twz (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
log:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
tmp-volume:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
local-bin:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
local-config:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
local-plugins:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
plugins:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-lupin59-master-0
ReadOnly: false
kubeblocks:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
tls:
Type: Secret (a volume populated by a Secret)
SecretName: lupin59-master-tls-certs
Optional: false
es-cm:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: lupin59-master-es-cm
Optional: false
scripts:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: lupin59-master-scripts
Optional: false
kube-api-access-k2twz:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
Optional: false
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 35m default-scheduler Successfully assigned default/lupin59-master-0 to olares
Normal Pulled 35m kubelet Container image "docker.io/library/elasticsearch:8.15.5" already present on machine
Normal Created 35m kubelet Created container: prepare-plugins
Normal Started 35m kubelet Started container prepare-plugins
Normal Pulled 35m kubelet Container image "docker.io/library/elasticsearch:8.15.5" already present on machine
Normal Created 35m kubelet Created container: install-plugins
Normal Started 35m kubelet Started container install-plugins
Normal Pulled 35m kubelet Container image "docker.io/apecloud/elasticsearch-agent:0.1.0" already present on machine
Normal Created 35m kubelet Created container: install-es-agent
Normal Started 35m kubelet Started container install-es-agent
Normal Pulled 35m kubelet Container image "apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/kubeblocks-tools:1.0.1" already present on machine
Normal Created 35m kubelet Created container: init-kbagent
Normal Started 35m kubelet Started container init-kbagent
Normal Pulled 35m kubelet Container image "docker.io/apecloud/curl-jq:0.1.0" already present on machine
Normal Created 35m kubelet Created container: kbagent-worker
Normal Started 35m kubelet Started container kbagent-worker
Normal Pulled 35m kubelet Container image "docker.io/library/elasticsearch:8.15.5" already present on machine
Normal Created 35m kubelet Created container: elasticsearch
Normal Started 35m kubelet Started container elasticsearch
Metadata
Metadata
Assignees
Labels
kind/bugSomething isn't workingSomething isn't working