Skip to content

[BUG] kafka separated stop -> start broker crash #9938

@JashBook

Description

@JashBook

Describe the bug
A clear and concise description of what the bug is.

helm get notes -n kb-system kb-addon-kafka  
NOTES:
CHART NAME: kafka
CHART VERSION: 1.1.0-alpha.0
APP VERSION: 3.3.2

KubeBlocks Kafka server cluster definition, start create your Kafka Server Cluster with following command:

    kbcli cluster create kafka


Release Information:
  Commit ID: "900fa110fb5bc1b3b09ebe48c35a22f9b6eb0a5c"
  Commit Time: "2025-12-03 14:43:50 +0800"
  Release Branch: "v1.1.0-alpha.5"
  Release Time:  "2025-12-15 09:39:54 +0800"
  Enterprise: "false"

➜  ~ 
➜  ~ kbcli version
Kubernetes: v1.30.4-vke.10
KubeBlocks: 1.1.0-alpha.5
kbcli: 1.0.2-beta.0

To Reproduce
Steps to reproduce the behavior:

  1. create cluster
apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: kafka-gornmp
  namespace: default
  annotations:
    "kubeblocks.io/extra-env": '{"KB_KAFKA_ENABLE_SASL":"false","KB_KAFKA_BROKER_HEAP":"-XshowSettings:vm -XX:MaxRAMPercentage=100 -Ddepth=64","KB_KAFKA_CONTROLLER_HEAP":"-XshowSettings:vm -XX:MaxRAMPercentage=100 -Ddepth=64","KB_KAFKA_PUBLIC_ACCESS":"false"}'
spec:
  clusterDef: kafka
  topology: separated_monitor
  terminationPolicy: Delete
  componentSpecs:
    - name: kafka-broker
      tls: false
      disableExporter: true
      replicas: 3
      serviceVersion: 3.3.2
      services:
        - name: advertised-listener
          serviceType: ClusterIP
          podService: true
      resources:
        requests:
          cpu: 500m
          memory: 1Gi
        limits:
          cpu: 500m
          memory: 1Gi
      env:
        - name: KB_BROKER_DIRECT_POD_ACCESS
          value: "false"
        - name: KB_KAFKA_ENABLE_SASL_SCRAM
          value: "false"
      volumeClaimTemplates:
        - name: data
          spec:
            storageClassName: 
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
        - name: metadata
          spec:
            storageClassName: 
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
    - name: kafka-controller
      tls: false
      replicas: 3
      serviceVersion: 3.3.2
      disableExporter: true
      resources:
        requests:
          cpu: 500m
          memory: 1Gi
        limits:
          cpu: 500m
          memory: 1Gi
      volumeClaimTemplates:
        - name: metadata
          spec:
            storageClassName: 
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
    - name: kafka-exporter
      replicas: 1
      disableExporter: true
      env:
        - name: KB_KAFKA_ENABLE_SASL_SCRAM
          value: "false"
      resources:
        requests:
          cpu: 500m
          memory: 1Gi
        limits:
          cpu: 500m
          memory: 1Gi

  1. stop
kbcli cluster stop kafka-gornmp --auto-approve 
OpsRequest kafka-gornmp-stop-nh8f8 created successfully, you can view the progress:
	kbcli cluster describe-ops kafka-gornmp-stop-nh8f8 -n default

kubectl get cluster 
NAME           CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS    AGE
kafka-gornmp   kafka                Delete               Stopped   2m22s
  1. start
kbcli cluster start kafka-gornmp 
  1. See error
kubectl get cluster kafka-gornmp 
NAME           CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS     AGE
kafka-gornmp   kafka                Delete               Abnormal   2m52s
➜  ~ 
➜  ~ kubectl get pod 
NAME                          READY   STATUS             RESTARTS      AGE
kafka-gornmp-kafka-broker-0   1/2     CrashLoopBackOff   1 (14s ago)   32s
kafka-gornmp-kafka-broker-1   1/2     CrashLoopBackOff   1 (14s ago)   32s
kafka-gornmp-kafka-broker-2   1/2     CrashLoopBackOff   1 (14s ago)   32s

logs error pod

kubectl logs kafka-gornmp-kafka-broker-0 --tail 50 --previous 
Defaulted container "kafka" out of: kafka, jmx-exporter, kafkatool (init)
convert prop 
line '' has no value; skipped
convert prop delegation.token.max.lifetime.ms=604800000
[cfg]export KAFKA_CFG_DELEGATION_TOKEN_MAX_LIFETIME_MS=604800000
convert prop delegation.token.expiry.time.ms=86400000
[cfg]export KAFKA_CFG_DELEGATION_TOKEN_EXPIRY_TIME_MS=86400000
convert prop delegation.token.expiry.check.interval.ms=3600000
[cfg]export KAFKA_CFG_DELEGATION_TOKEN_EXPIRY_CHECK_INTERVAL_MS=3600000
convert prop password.encoder.cipher.algorithm=AES/CBC/PKCS5Padding
[cfg]export KAFKA_CFG_PASSWORD_ENCODER_CIPHER_ALGORITHM=AES/CBC/PKCS5Padding
convert prop password.encoder.key.length=128
[cfg]export KAFKA_CFG_PASSWORD_ENCODER_KEY_LENGTH=128
convert prop password.encoder.iterations=4096
[cfg]export KAFKA_CFG_PASSWORD_ENCODER_ITERATIONS=4096
convert prop 
line '' has no value; skipped
convert prop 
line '' has no value; skipped
convert prop 
line '' has no value; skipped
convert prop 
line '' has no value; skipped
convert prop allow.everyone.if.no.acl.found=true
[cfg]export KAFKA_CFG_ALLOW_EVERYONE_IF_NO_ACL_FOUND=true
convert prop 
line '' has no value; skipped
convert prop 
line '' has no value; skipped
KAFKA_KRAFT_CLUSTER_ID=08e7ab09-a68f-4881-a9aa-50e2fef2835c
export KAFKA_KRAFT_CLUSTER_ID=08e7ab09-a68f-4881-a9a
find advertised_ports:kafka-gornmp-kafka-broker-advertised-listener-0:9092
find pod_name_ordinal:0
find svc_name:kafka-gornmp-kafka-broker-advertised-listener-0,port:9092,svc_name_ordinal:0
Found matching svcName and port for podName 'kafka-gornmp-kafka-broker-0', BROKER_ADVERTISED_PORT: kafka-gornmp-kafka-broker-advertised-listener-0:9092,kafka-gornmp-kafka-broker-advertised-listener-1:9092,kafka-gornmp-kafka-broker-advertised-listener-2:9092. svcName: kafka-gornmp-kafka-broker-advertised-listener-0, port: 9092.
[cfg]KAFKA_CFG_ADVERTISED_LISTENERS=INTERNAL://kafka-gornmp-kafka-broker-0.kafka-gornmp-kafka-broker-headless.default.svc.cluster.local:9094,CLIENT://kafka-gornmp-kafka-broker-0.kafka-gornmp-kafka-broker-headless.default.svc.cluster.local:9092
[cfg]KAFKA_CFG_NODE_ID=100
[cfg]export KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=,for kafka-broker.
[cfg]log to /bitnami/kafka/logs log4j configuration added.
kafka 10:10:14.67 
kafka 10:10:14.67 Welcome to the Bitnami kafka container
kafka 10:10:14.67 Subscribe to project updates by watching https://github.com/bitnami/containers
kafka 10:10:14.67 Submit issues and feature requests at https://github.com/bitnami/containers/issues
kafka 10:10:14.67 
kafka 10:10:14.67 INFO  ==> ** Starting Kafka setup **
kafka 10:10:14.69 DEBUG ==> Validating settings in KAFKA_* env vars...
kafka 10:10:14.70 ERROR ==> KRaft requires KAFKA_CFG_CONTROLLER_QUORUM_VOTERS to be set
kafka 10:10:14.70 WARN  ==> KAFKA_CFG_NODE_ID must match what is set in KAFKA_CFG_CONTROLLER_QUORUM_VOTERS
kafka 10:10:14.70 WARN  ==> KAFKA_CFG_PROCESS_ROLES must include 'controller' for KRaft
kafka 10:10:14.70 WARN  ==> KAFKA_CFG_LISTENERS must include a listener for CONTROLLER
kafka 10:10:14.70 WARN  ==> You set the environment variable ALLOW_PLAINTEXT_LISTENER=yes. For safety reasons, do not use this flag in a production environment.
kubectl logs kafka-gornmp-kafka-broker-1 --tail 20 --previous
Defaulted container "kafka" out of: kafka, jmx-exporter, kafkatool (init)
find pod_name_ordinal:1
find svc_name:kafka-gornmp-kafka-broker-advertised-listener-0,port:9092,svc_name_ordinal:0
find svc_name:kafka-gornmp-kafka-broker-advertised-listener-1,port:9092,svc_name_ordinal:1
Found matching svcName and port for podName 'kafka-gornmp-kafka-broker-1', BROKER_ADVERTISED_PORT: kafka-gornmp-kafka-broker-advertised-listener-0:9092,kafka-gornmp-kafka-broker-advertised-listener-1:9092,kafka-gornmp-kafka-broker-advertised-listener-2:9092. svcName: kafka-gornmp-kafka-broker-advertised-listener-1, port: 9092.
[cfg]KAFKA_CFG_ADVERTISED_LISTENERS=INTERNAL://kafka-gornmp-kafka-broker-1.kafka-gornmp-kafka-broker-headless.default.svc.cluster.local:9094,CLIENT://kafka-gornmp-kafka-broker-1.kafka-gornmp-kafka-broker-headless.default.svc.cluster.local:9092
[cfg]KAFKA_CFG_NODE_ID=101
[cfg]export KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=,for kafka-broker.
[cfg]log to /bitnami/kafka/logs log4j configuration added.
kafka 10:10:24.67 
kafka 10:10:24.67 Welcome to the Bitnami kafka container
kafka 10:10:24.67 Subscribe to project updates by watching https://github.com/bitnami/containers
kafka 10:10:24.67 Submit issues and feature requests at https://github.com/bitnami/containers/issues
kafka 10:10:24.67 
kafka 10:10:24.67 INFO  ==> ** Starting Kafka setup **
kafka 10:10:24.70 DEBUG ==> Validating settings in KAFKA_* env vars...
kafka 10:10:24.70 ERROR ==> KRaft requires KAFKA_CFG_CONTROLLER_QUORUM_VOTERS to be set
kafka 10:10:24.70 WARN  ==> KAFKA_CFG_NODE_ID must match what is set in KAFKA_CFG_CONTROLLER_QUORUM_VOTERS
kafka 10:10:24.70 WARN  ==> KAFKA_CFG_PROCESS_ROLES must include 'controller' for KRaft
kafka 10:10:24.70 WARN  ==> KAFKA_CFG_LISTENERS must include a listener for CONTROLLER
kafka 10:10:24.76 WARN  ==> You set the environment variable ALLOW_PLAINTEXT_LISTENER=yes. For safety reasons, do not use this flag in a production environment.
➜  ~ 
➜  ~ kubectl logs kafka-gornmp-kafka-broker-2 --tail 20 --previous
Defaulted container "kafka" out of: kafka, jmx-exporter, kafkatool (init)
find svc_name:kafka-gornmp-kafka-broker-advertised-listener-0,port:9092,svc_name_ordinal:0
find svc_name:kafka-gornmp-kafka-broker-advertised-listener-1,port:9092,svc_name_ordinal:1
find svc_name:kafka-gornmp-kafka-broker-advertised-listener-2,port:9092,svc_name_ordinal:2
Found matching svcName and port for podName 'kafka-gornmp-kafka-broker-2', BROKER_ADVERTISED_PORT: kafka-gornmp-kafka-broker-advertised-listener-0:9092,kafka-gornmp-kafka-broker-advertised-listener-1:9092,kafka-gornmp-kafka-broker-advertised-listener-2:9092. svcName: kafka-gornmp-kafka-broker-advertised-listener-2, port: 9092.
[cfg]KAFKA_CFG_ADVERTISED_LISTENERS=INTERNAL://kafka-gornmp-kafka-broker-2.kafka-gornmp-kafka-broker-headless.default.svc.cluster.local:9094,CLIENT://kafka-gornmp-kafka-broker-2.kafka-gornmp-kafka-broker-headless.default.svc.cluster.local:9092
[cfg]KAFKA_CFG_NODE_ID=102
[cfg]export KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=,for kafka-broker.
[cfg]log to /bitnami/kafka/logs log4j configuration added.
kafka 10:10:11.67 
kafka 10:10:11.67 Welcome to the Bitnami kafka container
kafka 10:10:11.67 Subscribe to project updates by watching https://github.com/bitnami/containers
kafka 10:10:11.67 Submit issues and feature requests at https://github.com/bitnami/containers/issues
kafka 10:10:11.67 
kafka 10:10:11.67 INFO  ==> ** Starting Kafka setup **
kafka 10:10:11.70 DEBUG ==> Validating settings in KAFKA_* env vars...
kafka 10:10:11.70 ERROR ==> KRaft requires KAFKA_CFG_CONTROLLER_QUORUM_VOTERS to be set
kafka 10:10:11.70 WARN  ==> KAFKA_CFG_NODE_ID must match what is set in KAFKA_CFG_CONTROLLER_QUORUM_VOTERS
kafka 10:10:11.70 WARN  ==> KAFKA_CFG_PROCESS_ROLES must include 'controller' for KRaft
kafka 10:10:11.70 WARN  ==> KAFKA_CFG_LISTENERS must include a listener for CONTROLLER
kafka 10:10:11.71 WARN  ==> You set the environment variable ALLOW_PLAINTEXT_LISTENER=yes. For safety reasons, do not use this flag in a production environment.

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions