Double starts while upgrading an existing broker with new setup.

[punit-kulal]

Foreword

We face multiple starts on the kafka cluster while setting bootstrapping a kafka broker with an existing log.dirs directory.

👻 Brief Description

Setup:

• An external disk acts as a persistent storage for log.dirs directory.
• A new VM with no installation of kafka.
• kafka.automatic_restart: true
  Bug:
• When the kafka recipe is run. Since, there's no kafka service running on the broker. kafka service resource starts up the kafka service.
• Since, automatic_restart: true, the service also is notified to perform a delayed restart.
• Since, the delayed restart is sent in close proximity to the start signal. The restart triggers an unclean shutdown on the kafka broker.

🥞 Cookbook version

Observed in version: 2.2.2, should be replicable in latest version as well.

👩‍🍳 Chef-Infra Version

12.96

Version of chef-client in your environment.

🎩 Platform details

Ubuntu 20.04
GCP

Steps To Reproduce

Steps to reproduce the behavior:

1. Shut down kafka broker.
2. Make some changes to any file monitored by automatic_restart flag, example server.properties.
3. Try running chef-client
4. Kafka starts up, and a SIGTERM is sent even before the start is complete.

🚓 Expected behavior

The kafka service should start only once. And should not be restarted.

Possible solution for 2.2.2:

• Initialise a node state in coordinate-kafka-start which points that the service is marked for restart.
• skip_start from the kafka_service_actions if that node state is set to true.
• I can raise a possible PR with the same solution.
• The solution is possible for v2.2.2 because the coordinate-kafka-start block is called immediately in v2.2.2