39-Monolith-to-Microservice-Migration.md

Moving from Monoliths to Microservices with Kafka Integration

Here's how you can migrate from a monolithic application to a microservice-based architecture, integrating Kafka for asynchronous communication between services.

Key Steps for Migration:

1. Contract:
   Define clear service contracts (API endpoints) for communication between microservices.
   • Example: Using REST API for synchronous service communication.

# Example: User Service - Flask (Monolith)
from flask import Flask, jsonify

app = Flask(__name__)

# Monolithic API for user management
@app.route("/users")
def get_users():
    return jsonify({"users": ["Alice", "Bob", "Charlie"]})

if __name__ == "__main__":
    app.run(debug=True)

2. Router:
   Use an API Gateway (like Kong, Nginx) to route requests to appropriate microservices.

# Example: Nginx API Gateway routing config
http {
    server {
        listen 80;
        
        location /user {
            proxy_pass http://user-service:5000;
        }
        
        location /product {
            proxy_pass http://product-service:5000;
        }
    }
}

3. Simplify Deployments:
   Use CI/CD pipelines for seamless deployments of individual services (Docker, Kubernetes).

# Example: Simple CI/CD pipeline with GitLab CI
stages:
  - build
  - test
  - deploy

deploy:
  stage: deploy
  script:
    - docker build -t user-service .
    - kubectl apply -f user-service-deployment.yaml

4. Communication:
   Choose Kafka for asynchronous communication between services.

• Producer (User Service) - Sends an event to Kafka when a user is created:

# User Service - Kafka Producer (Python)
from kafka import KafkaProducer
import json

# Kafka producer setup
producer = KafkaProducer(bootstrap_servers='localhost:9092', value_serializer=lambda v: json.dumps(v).encode('utf-8'))

# Send a message to Kafka topic 'user-created'
def send_message(user_data):
    producer.send('user-created', user_data)
    producer.flush()

# Example usage: 
send_message({"user": "Alice", "action": "created"})

• Consumer (Order Service) - Consumes events from Kafka to react to user creation:

# Order Service - Kafka Consumer (Python)
from kafka import KafkaConsumer
import json

# Kafka consumer setup
consumer = KafkaConsumer('user-created', bootstrap_servers='localhost:9092', group_id='order-service', value_deserializer=lambda x: json.loads(x.decode('utf-8')))

# Process messages
for message in consumer:
    user_data = message.value
    print(f"New user created: {user_data['user']}")
    # Logic to create order based on new user

5. Logging:
   Centralized logging for distributed microservices using ELK stack (ElasticSearch, Logstash, and Kibana) for tracking service logs.

# Example: Docker Compose for ELK Stack
version: '3'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.10.0
    environment:
      - discovery.type=single-node
  logstash:
    image: docker.elastic.co/logstash/logstash:7.10.0
    ports:
      - "5044:5044"
  kibana:
    image: docker.elastic.co/kibana/kibana:7.10.0
    ports:
      - "5601:5601"

---

Summary of Steps to Migrate:

1. Define Service Boundaries: Split monolith into independent services (e.g., user service, product service, order service).
2. Start with One Service: Migrate the user management functionality into a microservice.
3. Gradual Refactoring: Migrate other modules (e.g., product, order) one at a time.
4. Data Management: Use Kafka for managing data consistency and event-driven communication across services.
5. Monitor and Scale: Add centralized logging and monitoring with ELK Stack.

---

Example of Microservices Architecture with Kafka:

• Monolithic Application: A single Python Flask app handles users, products, and orders.
• Microservices Architecture:
  • User Service: Manages users.
  • Product Service: Manages product catalog.
  • Order Service: Handles orders and depends on user service.

# Example: Microservices with Docker Compose
version: '3'
services:
  user-service:
    image: user-service:latest
    ports:
      - "5000:5000"
  product-service:
    image: product-service:latest
    ports:
      - "5001:5001"
  order-service:
    image: order-service:latest
    ports:
      - "5002:5002"
  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_LISTENER: INSIDE:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL: PLAINTEXT
      KAFKA_LISTENER_NAME_INTERNAL: INSIDE
      KAFKA_LISTENER_INTERNAL: INSIDE://kafka:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"

---

Benefits of Kafka in Microservices:

1. Asynchronous Communication: Decouples services and allows for efficient communication without blocking.
2. Scalability: Kafka’s high throughput and partitioning make it suitable for large-scale distributed systems.
3. Fault Tolerance: Kafka retains messages, allowing services to recover and reprocess events if necessary.

By integrating Kafka in the migration process, we ensure better scalability, reliability, and fault tolerance in our distributed microservices architecture.