🚀 Distributed Log Analytics Engine (Java)

A scalable backend-style Java application that simulates a distributed log analytics system.
It ingests raw log files, processes events, computes metrics, detects anomalies, and persists results.

Built using:

• Core Java
• Multi-package architecture
• File I/O
• Stream API
• Thread-safe persistence

This project demonstrates real backend engineering practices such as modular design, analytics pipelines, and alerting systems.

---

🧠 Features

✔ Log ingestion from files
✔ Event parsing into structured objects
✔ Metrics calculation (INFO / WARN / ERROR counts)
✔ Anomaly detection with alerting
✔ Thread-safe result persistence
✔ Clean package-based architecture

---

📂 Project Structure

distributed-log-engine/ │ ├── .vscode/ │ └── settings.json │ ├── src/ │ └── com/logengine/ │ ├── app/ → Application entry point │ ├── analytics/ → Metrics & anomaly detection │ ├── alert/ → Alert services │ ├── model/ → Data models │ ├── parser/ → Log ingestion │ ├── processor/ → Processing pipeline │ └── persistence/ → Storage layer │ ├── data/ │ ├── sample-logs.txt │ └── analytics-results.txt │ └── README.md

---

▶️ How to Run

1️⃣ Compile the project

From the project root:

rm -rf bin
mkdir bin
javac -d bin $(find src -name "*.java")

2️⃣ Run the application
java -cp bin com.logengine.app.Main

📝 Sample Input (data/sample-logs.txt)
INFO Application started
WARN Memory usage high
ERROR Null pointer exception
ERROR Database timeout
ERROR Disk failure
INFO User logged in

📊 Example Output
🚀 Distributed Log Analytics Engine Starting...
📊 Metrics Summary:
INFO  : 2
WARN  : 1
ERROR : 3
🚨 ALERT: High error rate detected: 3
✅ Done. Check data/analytics-results.txt


Results are saved to:

data/analytics-results.txt

🧩 Architecture Overview
🔹 LogParser

Reads raw log files and returns lines.

🔹 LogProcessor

Transforms raw strings into structured LogEvent objects.

🔹 MetricsCalculator

Computes analytics metrics (INFO/WARN/ERROR counts).

🔹 AnomalyDetector

Detects high error rates and triggers alerts.

🔹 AlertService

Interface for alerting implementations (console-based for now).

🔹 ResultStore

Persists metrics and logs using thread-safe file writing.

🧪 Technologies Used

Java 17+

Java Streams

File I/O (NIO)

Concurrency (ReentrantLock)

Modular package structure

VS Code Java tooling

🚀 Why This Project Matters

This project demonstrates:

Backend data pipelines

Analytics processing

Alerting systems

Clean object-oriented design

Real-world logging workflows

It reflects patterns used in:

Distributed systems

Observability platforms

Log monitoring tools

Enterprise backend services