High Volume CSV Ingestion Service

A Spring Boot microservice for high-volume CSV data ingestion using Kafka streaming and PostgreSQL batch insertion. Built with Java 21 and designed for efficient processing of large datasets.

Quick Start

Setup

1. Configure Environment Variables

   cp .env.template .env

   Edit .env and set your BASE_DIR (absolute path) and database credentials.

2. Pull Pre-built Docker Image (NOT NECESSARY , JUST AS AN INFO)

   docker pull robicodex18/highvolumecsvingestionservice:1.0.0

3. Start Services

   docker-compose up -build

Query API

Execute SQL queries against the ingested data:

curl -X POST http://localhost:8080/api/query \
  -H "Content-Type: text/plain" \
  -d "SELECT * FROM your_table LIMIT 10"

Architecture

• Check Architecture.jpg

The service uses a streaming architecture:

• Kafka for message queuing and data streaming
• PostgreSQL for persistent storage with batch insertion optimization
• Spring Boot for REST API and service orchestration

Check .jpg file for Architecture diagram

Development

Build from source:

docker-compose up --build

Project Details .md

• Check HighVolumeIngestionService.md
• You can check PDF also