Kafka Consumer Lag calculation

Consumer lag is one of the most important metrics for estimating the load in event driven systems. This POC generates a lag between producer and consumer, another API calculates the generated lag.

How to run

1. Install docker and setup kafka using the kafka_setup.yml docker compose file.

   docker-compose -f .\kafka_setup.yml up -d

2. Get into the kafka container as root.

   docker exec -it kafka bash

3. Get into the corresponding kafka folder.

   cd opt/kafka_2.13-2.8.1/bin

4. Create a topic of any name.

   kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic <topic_name>

5. Start producer from the main file. Please edit the application.properties file as per your requirements.
6. Start consumer from the main file. Please edit the application.properties file as per your requirements. It is preferred that consumer wait time should be more than producer wait time.
7. Start lag analyzer service.
8. Goto the below url where 'temp_group' is the "group_name" of your choice. (Should match with consumer properties)

   http://localhost:8080/lag/temp_group

Output

1. Producer Output

   

2. Consumer Output

   

3. Lag initial stages

   

4. Lag increasing in later stages

   

Additional commands

docker-compose -f .\kafka_setup.yml up -d

cd opt/kafka_2.13-2.8.1/bin

kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic temp_topic

kafka-topics.sh --list --zookeeper zookeeper:2181

kafka-consumer-groups.sh --bootstrap-server kafka:9092 --describe --group temp_group

kafka-console-producer.sh --broker-list kafka:9092 --topic temp_topic

kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic temp_topic --from-beginning --group temp_group