Kafka-powered microservice pipeline processing 500K+ daily events with circuit-breaker patterns, idempotency guarantees, and Prometheus/CloudWatch observability. Achieves 3× throughput improvement (500 → 1,500 req/s) through producer batching, manual consumer ACK, and partitioned event routing.
┌─────────────────────────────────────────────────────────────────────────────┐
│ Event-Driven Order Pipeline │
└─────────────────────────────────────────────────────────────────────────────┘
REST Client
│
▼
┌──────────────┐ POST /api/v1/orders
│ OrderController│─────────────────────────────────────┐
└──────────────┘ │
│ ▼
│ ┌──────────────┐
│ │ OrderService │
│ │ (+ Circuit │
│ │ Breaker) │
│ └──────┬───────┘
│ │ save()
│ ▼
│ ┌──────────────┐
│ │ PostgreSQL │
│ │ (orders) │
│ └──────────────┘
│ │
│ │ publish event
│ ▼
│ ┌───────────────────────┐
│ │ OrderEventProducer │
│ │ (idempotent producer │
│ │ key=orderId) │
│ └───────────┬───────────┘
│ │
│ ▼
│ ┌─────────────────────────┐
│ │ Kafka Topic │
│ │ order.events.v1 │
│ │ [partition 0│1│2] │
│ └────────────┬────────────┘
│ │ │ on 3× failure
│ │ ▼
│ │ ┌──────────────┐
│ │ │ .DLT topic │
│ │ │ (dead letter)│
│ │ └──────────────┘
│ │
│ ▼
│ ┌─────────────────────────┐
│ │ OrderEventConsumer │
│ │ (3 concurrent threads)│
│ │ manual ACK + retry │
│ └────────────┬────────────┘
│ │
│ ┌───────────────┼───────────────┐
│ ▼ ▼ ▼
│ ORDER_CREATED PAYMENT_VALIDATED INVENTORY_RESERVED
│ │ │ │
│ validatePayment reserveInventory shipOrder
│
│ ┌──────────────────────────────────────────────────────┐
│ │ Supporting Services │
│ │ │
│ │ ┌───────────────┐ ┌──────────────────────┐ │
│ │ │ Redis Cache │ │ Micrometer/Prometheus│ │
│ │ │ (Idempotency │ │ /actuator/prometheus │ │
│ │ │ key store) │ │ → CloudWatch │ │
│ │ └───────────────┘ └──────────────────────┘ │
│ └──────────────────────────────────────────────────────┘
PENDING ──────────────► PAYMENT_VALIDATED ──► INVENTORY_RESERVED ──► SHIPPED ──► DELIVERED
│ │ │
│ (circuit open) (stock out)
▼ ▼ ▼
FAILED ◄────────────────────────────────────────────────
│
(retry >= 3)
▼
DEAD_LETTERED
| Concern | Solution | Why |
|---|---|---|
| Duplicate events | Redis SET NX idempotency keys (24h TTL) |
Kafka at-least-once delivery can replay — guard all consumers |
| Partition ordering | Producer key = orderId |
All events for one order land on same partition → sequential processing |
| Cascading failures | Resilience4j circuit breaker (50% threshold, 30s open) | Prevent payment service downtime from taking down the pipeline |
| Throughput | 32KB batch, 5ms linger, snappy compression | 3× req/s improvement without topology changes |
| Observability | Micrometer counters per event type + Prometheus scrape | Feeds CloudWatch SLO alerts; MTTR < 2 min |
| Consumer safety | Manual ACK, DLQ after 3 retries | No silent message loss on pod restart or rebalance |
| Layer | Technology |
|---|---|
| Language | Java 17 |
| Framework | Spring Boot 3.2 |
| Messaging | Apache Kafka (Confluent 7.6) |
| Database | PostgreSQL 16 + Flyway migrations |
| Cache | Redis 7.2 (idempotency store) |
| Circuit Breaker | Resilience4j 2.2 |
| Observability | Micrometer + Prometheus + CloudWatch |
| Testing | JUnit 5, Mockito, Testcontainers |
| Coverage | JaCoCo (75% minimum enforced) |
| Container | Docker + docker-compose |
| CI/CD | GitHub Actions |
- Docker & Docker Compose
- Java 17+
- Maven 3.9+
docker-compose up -d zookeeper kafka postgres redisWait for all services to be healthy (check with docker-compose ps).
./mvnw spring-boot:runOr with Docker:
docker-compose up --build app| Service | URL |
|---|---|
| API | http://localhost:8080/api/v1/orders |
| Swagger UI | http://localhost:8080/swagger-ui.html |
| Actuator / Health | http://localhost:8080/actuator/health |
| Prometheus Metrics | http://localhost:8080/actuator/prometheus |
| Kafka UI | http://localhost:8090 |
POST /api/v1/orders
Content-Type: application/json
{
"customerId": "cust-123",
"productId": "prod-456",
"quantity": 2,
"totalAmount": 99.99
}GET /api/v1/orders/{orderId}POST /api/v1/orders/{orderId}/validate-paymentGET /api/v1/orders/customer/{customerId}# Unit tests only
mvn test
# Unit + coverage check (75% threshold enforced)
mvn verify -Ptest
# Integration tests (requires Docker for Testcontainers)
mvn verify -Pintegration-testThe application exposes the following custom metrics at /actuator/prometheus:
| Metric | Description |
|---|---|
orders_created_total |
Total orders ingested |
orders_payment_validated_total |
Successful payment validations |
orders_shipped_total |
Orders successfully shipped |
orders_dead_lettered_total |
Events exhausted retries → DLQ |
kafka_events_duplicate_skipped_total |
Idempotency guards triggered |
kafka_publish_failure_total |
Producer send failures |
circuit_breaker_open_total |
Circuit breaker trips |
src/
├── main/java/com/chandu/orderapi/
│ ├── config/ # Kafka producer/consumer, Redis, Resilience4j config
│ ├── controller/ # REST endpoints with OpenAPI annotations
│ ├── consumer/ # Kafka listener with idempotency + manual ACK
│ ├── domain/ # Order entity, OrderEvent DTO, OrderStatus enum
│ ├── exception/ # Custom exceptions + GlobalExceptionHandler
│ ├── metrics/ # Micrometer custom counters
│ ├── producer/ # Kafka template wrapper with async callbacks
│ ├── repository/ # Spring Data JPA queries
│ └── service/ # Business logic, circuit breaker, state transitions
├── main/resources/
│ ├── db/migration/ # Flyway SQL migrations
│ └── application.yml # Config for local + docker profiles
└── test/java/... # Unit tests (JUnit 5 + Mockito, 75%+ coverage)
Sai Chandu Vallaboju — Senior Software Engineer
GitHub · LinkedIn · chanduvallaboju01@gmail.com