SmartGridMonitoring

SmartGridMonitoring is a Scala-based prototype for monitoring transformer telemetry in a distributed smart grid. It combines a streaming alert pipeline with a batch analytics layer, using Kafka for event transport, PostgreSQL for alert persistence, and an S3-compatible data lake for historical processing.

The project is built as a small collection of independent services so each part of the pipeline can be developed, run, and tested separately.

Architecture

The system follows two complementary flows:

• Real-time monitoring: simulated transformer sensors publish telemetry to Kafka. A detector service consumes the stream, evaluates operational thresholds, and emits alerts when a transformer looks overloaded or at risk.
• Historical analytics: telemetry is written into a data lake, transformed through Bronze, Silver, and Gold layers, then aggregated into statistics that can be used for reporting or future forecasting work.

Core infrastructure is defined in docker-compose.yml:

• Kafka cluster for telemetry and alert topics
• PostgreSQL for alert storage
• MinIO as the local S3-compatible object store

Components

Module	Purpose
shared	Shared telemetry models, alert models, and threshold definitions
sensor-simulator	Kafka producer that generates transformer telemetry
alert-detector	Streaming consumer that detects risky transformer states
alert-handler	Alert API, dashboard rendering, notification handling, and PostgreSQL persistence
bronze-ingestor	Kafka consumer that writes raw telemetry to the data lake
datalake	Spark job that builds cleaned and structured lake layers
analytics	Spark aggregation job that produces reporting datasets and an HTML analytics page

Data Flow

sensor-simulator
    -> Kafka telemetry topic
        -> alert-detector
            -> Kafka alert topic
                -> alert-handler
                    -> PostgreSQL / notifications

Kafka telemetry topic
    -> bronze-ingestor
        -> MinIO Bronze layer
            -> datalake
                -> MinIO Silver and Gold layers
                    -> analytics
                        -> statistics and HTML report

Run Locally

Start the replicated Kafka cluster, PostgreSQL, and MinIO:

docker compose up -d kafka kafka2 kafka3 postgres minio
docker compose run --rm minio-init

Start the streaming services in separate terminals:

sbt "simulator/run"
sbt "alertDetector/run"
SMART_GRID_MAIL_MODE=file sbt "alertHandler/run"

Write a batch of telemetry events to the Bronze layer:

BRONZE_MAX_MESSAGES=20 sbt "bronzeIngestor/run"

Build the Silver and Gold lake layers:

sbt "datalake/run"

Generate analytics outputs:

sbt "analytics/run"

Bronze, Silver, Gold, and statistics are written to the MinIO bucket smartgrid-lake under s3a://smartgrid-lake/. The analytics page is written to s3a://smartgrid-lake/stats/index.html.

Development Notes

The root SBT build aggregates all modules, while each runnable component keeps its own Main.scala and build.sbt so it can also be opened or run independently.

Generated build outputs, local data lake files, logs, and local environment files are intentionally excluded from Git.