Invalid cluster.id in: /var/lib/kafka/data-0/kafka-log0/meta.properties

[nlaimutyte]

Hi, I use Strimzi Kafka operator as dependency in my Helm chart. No configurations changed.
I also use template from example https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/kafka/kraft/kafka.yam without changes.

I previously used 0.40.0 operator version with 3.7.0 Kafka version. I didn't have problems.
Now I updated to 0.44.0 with Kafka version 3.8.0. Also tried 0.43.0 operator version with 3.8.0 Kafka version. Got the same results.

I install helm chart. Then uninstall it. Leave all PV's and PVC's. Then try to install my Helm chart again.
All brokers and controllers fails to start. In logs I see error:

Exception in thread "main" java.lang.RuntimeException: Invalid cluster.id in: /var/lib/kafka/data-0/kafka-log0/meta.properties. Expected L8HFn0aZQdyZl5Bdqphv4Q, but read tELt6NAdRXeyu4wb-QTT5A
        at org.apache.kafka.metadata.properties.MetaPropertiesEnsemble.verify(MetaPropertiesEnsemble.java:509)
        at kafka.tools.StorageTool$.formatCommand(StorageTool.scala:531)
        at kafka.tools.StorageTool$.runFormatCommand(StorageTool.scala:140)
        at kafka.tools.StorageTool$.execute(StorageTool.scala:80)
        at kafka.tools.StorageTool$.main(StorageTool.scala:53)
        at kafka.tools.StorageTool.main(StorageTool.scala)

Is it bug?
Do I need to change configurations?

Discussion https://github.com/orgs/strimzi/discussions/10082 is kind of related.

[scholzj]

This looks like you deleted your cluster and then tried to reuse the PVs/PVCs. That does not work so easily and Kafka in KRaft mode protects itself against it. If it is something you really want to do, you should make sure the first set the correct cluster ID (the one you see in the error) in the status section of the Kafka CR. If you really wanted to delete the cluster and start a new one, you should delete the PVCs (and PVs if desired) before starting the new cluster.

[arielmoraes]

@scholzj, what is the solution for unexpected restarts? Can I deploy the first time with a clusterId set? If the PVC is already created, why is it not trying to pick up the old clusterId?