Kafka Cluster with KRaft Mode

This repository contains configuration for setting up a highly available Kafka cluster using Docker Compose. The setup leverages KRaft mode (Kafka without ZooKeeper) and includes Kafdrop for cluster monitoring.

Architecture Overview

The architecture establishes a three-node Kafka cluster in KRaft mode with separated communication channels for different traffic types:

Node Structure

• kafka-1 (Node 1001)

  • Dual role: Broker + Controller
  • CLIENT listener: localhost:9092
  • INTERNAL listener: kafka-1:19092
  • CONTROLLER listener: kafka-1:29092

• kafka-2 (Node 1002)

  • Dual role: Broker + Controller
  • CLIENT listener: localhost:9093
  • INTERNAL listener: kafka-2:19093
  • CONTROLLER listener: kafka-2:29093

• kafka-3 (Node 1003)

  • Single role: Broker only
  • CLIENT listener: localhost:9094
  • INTERNAL listener: kafka-3:19094

Communication Channels

1. Client-to-Broker Communication

• External applications connect to any broker via CLIENT listeners
• Connection addresses: localhost:9092, localhost:9093, localhost:9094
• Uses PLAINTEXT protocol (unencrypted)

2. Inter-Broker Communication

• Brokers communicate with each other via INTERNAL listeners
• Handles data replication, partition syncing, and leader elections
• Uses Docker network hostnames (kafka-1, kafka-2, kafka-3)
• Specified by KAFKA_CFG_INTER_BROKER_LISTENER_NAME=INTERNAL

3. Controller Communication

• Only between nodes 1001 and 1002 (controller nodes)
• Uses dedicated CONTROLLER listeners
• Manages KRaft consensus for cluster metadata
• Controller quorum voters: 1001@kafka-1:29092, 1002@kafka-2:29093

4. Monitoring

• Kafdrop provides web-based monitoring on port 9000
• Connects to all brokers using their INTERNAL listener addresses
• Provides cluster topology, consumer groups, and topic inspection

Environment Variables Explained

Core Kafka Configuration

1. KAFKA_ENABLE_KRAFT: Enables the Kafka cluster to be bootstrapped in KRaft mode (without ZooKeeper).
2. ALLOW_PLAINTEXT_LISTENER: Enables plain-text communication (used during development).
3. KAFKA_CFG_NODE_ID: The unique ID of each Kafka broker.
4. KAFKA_KRAFT_CLUSTER_ID: The unique ID of the cluster, consistent across all brokers.

Role and Process Configuration

5. KAFKA_CFG_PROCESS_ROLES: Defines whether the node acts as a broker, controller, or both.
6. KAFKA_CFG_CONTROLLER_LISTENER_NAMES: Specifies the CONTROLLER listener used for KRaft controller communication.
7. KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: Maps each listener to its security protocol.
8. KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: Lists all controller nodes that participate in voting for cluster management decisions.

Listener Configuration

9. KAFKA_CFG_ADVERTISED_LISTENERS: Specifies the addresses that Kafka advertises to clients and other brokers.
10. KAFKA_CFG_LISTENERS: Defines where Kafka actually listens for connections.
11. KAFKA_CFG_INTER_BROKER_LISTENER_NAME: Specifies that brokers should communicate with each other using the INTERNAL listener.

Benefits of This Architecture

1. High Availability: Three broker nodes ensure data redundancy and fault tolerance
2. Separation of Traffic: Different listener types separate client, broker, and controller traffic
3. KRaft Mode: Elimination of ZooKeeper dependency simplifies the architecture
4. Monitoring: Kafdrop provides easy visibility into the cluster state
5. Fault Tolerance: Dual controller nodes ensure cluster management remains available if one controller fails

Getting Started

1. Run the cluster:

   docker-compose up -d

2. Access Kafdrop monitoring UI:

   http://localhost:9000
   

3. Connect to Kafka using any of the advertised listeners:

   localhost:9092, localhost:9093, or localhost:9094
   

Production Considerations

This setup uses PLAINTEXT protocol for simplicity. For production environments, consider:

1. Implementing SSL/TLS encryption
2. Adding authentication mechanisms
3. Adjusting memory and CPU limits
4. Implementing proper volume mounts for data persistence
5. Setting up monitoring and alerting beyond Kafdrop