Kafka Schema Registry Spring Boot Demo (Avro Producer & Consumer with PostgreSQL)

This repository demonstrates how to build an event-driven Spring Boot microservice using Apache Kafka, Confluent Schema Registry, Avro serialization and PostgreSQL persistence.

---

What this Demo has

This repository provides a complete Spring Boot example using Apache Kafka, Confluent Schema Registry and Avro serialization.

It demonstrates a Kafka Avro producer and consumer with PostgreSQL persistence, showing how schemas are registered, evolved and consumed in a real-world event-driven application.

---

What This Demo Includes

• Spring Boot Kafka producer using Avro
• Spring Boot Kafka consumer deserializing Avro messages
• Confluent Schema Registry integration
• PostgreSQL persistence using Spring Data JPA
• Schema evolution with backward compatibility
• Local development using Docker Compose

---

Who Should Use This Project

This demo is useful for developers who want to:

• Learn Kafka Schema Registry with Spring Boot
• Understand Avro serialization in Kafka
• Build producer/consumer pipelines with PostgreSQL
• Create a local Kafka development environment

---

✨ Key Features

• Kafka producer & consumer written in Java 21 with Spring Boot.
• Confluent Schema Registry integration with Avro serialization/deserialization.
• PostgreSQL persistence using Spring Data JPA; includes ready‑made table schema (users.contact).
• Schema evolution demonstrated with backward compatibility and versioning.
• Docker Compose configuration to spin up Kafka, Schema Registry and PostgreSQL locally.
• Confluent Cloud support (bring your own API keys) via the cloud Spring profile.

---

🏗️ Architecture Overview

                            ┌─────────────┐
                            │  Postman    │
                            │  (client)   │
                            └──────┬──────┘
                                   │ HTTP POST /users
                                   ▼
                        ┌─────────────────────┐
                        │ Spring Boot Producer│
                        │ (REST + Avro)       │
                        └─────────┬───────────┘
                                  Kafka topic (users.v1)
                        ┌─────────▼───────────┐
                        │   Kafka Brokers     │
                        │   + Schema Registry │
                        └─────────┬───────────┘
                                   │ Avro → User
                                   ▼
                        ┌─────────────────────┐
                        │ Spring Boot Consumer│
                        │ (Avro + JPA)        │
                        └─────────┬───────────┘
                                   │
                                   ▼
                              PostgreSQL

1. The producer exposes a /users REST endpoint. It receives a UserCreateRequest, validates it and converts it to an Avro User record. The record is serialized and published to Kafka.
2. The Schema Registry stores Avro schemas and enforces compatibility rules when new versions are registered.
3. The consumer listens to the users.v1 topic, deserializes Avro messages and maps them to a JPA UserEntity. It persists each user to the users.contact table in PostgreSQL.

This separation of concerns ensures loose coupling between services and safe schema evolution. The architecture represents a typical event-driven microservice using Kafka, Schema Registry and a relational database.

---

🗄️ Database Schema

The consumer stores events in a table named contact under the users schema. The DDL is:

DROP table IF EXISTS users.Contact;
CREATE SCHEMA IF NOT EXISTS users;

CREATE TABLE users.contact (
	id int8 GENERATED BY DEFAULT AS IDENTITY( INCREMENT BY 1 MINVALUE 1 MAXVALUE 9223372036854775807 START 1 CACHE 1 NO CYCLE) NOT NULL,
	userid text NULL,
	email text NULL,
	phone text NULL,
	first_name text NULL,
	last_name text NULL,
	is_active bool DEFAULT true NOT NULL,
	created_at text NULL,
	age int4 NULL,
	CONSTRAINT contact_pkey PRIMARY KEY (id)
);
CREATE INDEX idx_users_created_at ON users.contact USING btree (created_at DESC);
CREATE INDEX idx_users_email ON users.contact USING btree (email);
CREATE INDEX idx_users_is_active ON users.contact USING btree (is_active);

INSERT INTO users.Contact (id, userId, email, phone, first_name, last_name, is_active, created_at, age)
VALUES
  ('1',
   'u-22',
   'mstauroy@gmail.com',
   '2109456738',
   'Manthos',
   'Staurou',
   TRUE,
   '2025-10-19T21:00:00Z',
   35)
ON CONFLICT (id) DO NOTHING;


The JPA `UserEntity` maps to this table and uses a generated `id` as the primary key.  The `userid` column stores the logical identifier coming from the Avro record (`id` field).  You can customise this schema as needed.

---

### Start infrastructure

The project includes a Docker Compose setup for running:
- Apache Kafka
- Confluent Schema Registry
- PostgreSQL

Start Kafka, Schema Registry and PostgreSQL using Docker Compose:

```bash
# spin up Kafka, Schema Registry and Postgres
docker compose -f docker-compose.yml up -d

# Kafka will be available on localhost:29092
# Schema Registry on http://localhost:8081
# PostgreSQL on localhost:5432 (user: kafka / password: kafkaConfluent)

---

🐳 Full Local Stack (Kafka + Schema Registry + PostgreSQL)

For a complete local development environment, this project provides a docker-compose.yml file that starts:

• Apache Kafka
• Zookeeper
• Confluent Schema Registry
• PostgreSQL (used by the Kafka consumer)

---

Start the full stack

docker compose -f docker-compose.yml up -d

---
### Run the consumer

```bash
# from the project root
cd consumer-app
../mvnw spring-boot:run || mvn spring-boot:run

# The application runs on port 8089 by default and listens to the `users.v1` topic.

---

Run the producer

cd ../producer-app
../mvnw spring-boot:run || mvn spring-boot:run

# The application runs on port 8080 by default.  It exposes a POST /users endpoint.

---

Produce a user event

Send a HTTP POST to create a user:

curl -X POST http://localhost:8080/users \
  -H "Content-Type: application/json" \
  -d '{
    "id": "u-20",
    "email": "mstauroy@gmail.com",
    "phone": "2109456738",
    "firstName": "Manthos",
    "lastName": "Staurou",
    "isActive": true,
    "age": 35
  }'

You should see logs in the consumer indicating that the record was received and saved to PostgreSQL.

---

📸 Demo Screenshots

Below are screenshots of the end‑to‑end flow:

Producer logs

This log shows the producer publishing a user to the topic users.v1 using the Schema Registry and Avro serializer.

Consumer logs

This log shows the consumer subscribing to users.v1, consuming the Avro record and persisting it to the users.contact table in PostgreSQL.

Postman request

Use the provided Postman collection to test the API easily. The screenshot below shows the request body when creating a user via Postman.

☁️ Running with Confluent Cloud

To run against Confluent Cloud, create an account at confluent.cloud and provision a Kafka cluster and Schema Registry. Then set the following environment variables (either in a .env file or exported in your shell):

export CLOUD_BOOTSTRAP_SERVERS="pkc-xxxxx.us-central1.gcp.confluent.cloud:9092"
export CLOUD_API_KEY="<your-kafka-api-key>"
export CLOUD_API_SECRET="<your-kafka-api-secret>"
export SR_URL="https://xxxxx.us-central1.gcp.confluent.cloud"
export SR_API_KEY="<your-schema-registry-api-key>"
export SR_API_SECRET="<your-schema-registry-api-secret>"

Then start the applications with the cloud profile:

# Consumer
cd consumer-app
../mvnw spring-boot:run -Dspring-boot.run.profiles=cloud

# Producer
cd ../producer-app
../mvnw spring-boot:run -Dspring-boot.run.profiles=cloud

With this profile enabled, the applications will connect to Confluent Cloud using SASL/SSL and will not automatically register schemas; instead they will use the latest registered schema version.

---

🔄 Schema Evolution

Avro schemas can evolve while maintaining compatibility. To test schema evolution:

1. Modify common-schemas/src/main/avro/User.avsc (e.g., add an optional field with a default).
2. Build the common-schemas module (mvn install) to generate new Java classes.
3. Register the new schema version in Schema Registry (via the UI or API). Set the compatibility mode to BACKWARD or FULL for the subject users-value.
4. Deploy your producer and consumer using the new jar.

Because the services are configured with auto.register.schemas=false and use.latest.version=true (in cloud mode), producers will not register schemas automatically in higher environments. This encourages CI/CD pipelines to manage schemas explicitly.

---

🧪 Testing & CI

• Unit tests: Add tests for your mapper, service and controller layers using JUnit and Mockito.
• Integration tests: Use Testcontainers to spin up Kafka, Schema Registry and PostgreSQL in Docker for reproducible integration tests.
• Continuous Integration: Configure GitHub Actions or your preferred CI to run mvn test and build the Docker images.

---

🧭 About / Description

This project is a learning template and production starter kit for event‑driven architectures. It shows how to:

• Define strong data contracts using Avro and Schema Registry.
• Produce and consume Kafka events with Spring Boot.
• Persist events to a relational database (PostgreSQL).
• Evolve schemas safely and manage compatibility in CI/CD.

Feel free to fork, star and extend it for your own microservice projects.

---

📬 Postman Collection

A Postman collection is provided in the postman/ directory. Import it into Postman to quickly test the REST API:

• postman import kafka-schema-registry-spring-demo.postman_collection.json

The collection includes the Create User request with the correct JSON payload. Run it after starting the producer and consumer to see end‑to‑end functionality.

🤝 Contributing & Feedback

Contributions, feedback and issues are welcome! Feel free to open pull requests or issues. If you find this project helpful, please ⭐ star it on GitHub and share it on social media – it helps others discover it.