Add financial streaming dashboard for real-time pipeline

[AbdoMis]

Real-Time Analytics Pipeline & Predictive Terminal Implementation + 🎥 Video demo

This Pull Request integrates the core Real-Time Streaming & Analytics Engine into the FinanceLake platform. It establishes the complete data lifecycle—from ingestion to visualization—implementing the Medallion Architecture (Bronze/Silver/Gold layers) on Delta Lake.

Self-Contained Architecture: This contribution is designed as a fully self-contained module within the financelake_terminal/ directory. It maintains its own isolated configuration (.env) and dependencies (requirements.txt). This design choice ensures:

• Portability: The module can be deployed independently.
• Isolation: It prevents dependency conflicts with the root repository or other modules.

The primary objective is to enable low-latency market analysis and predictive modeling using open-source distributed computing technologies.

Synthesis of Contribution

1. Ingestion Layer (High-Frequency Data Stream)

• Component: producer.ipynb
• Description: Implemented a robust Kafka Producer that interfaces with the Finnhub WebSocket API.
• Functionality:
  • Subscribes to real-time trade ticks for major technology equities (e.g., AAPL, TSLA, NVDA).
  • Serializes trade data into JSON format.
  • Publishes payloads to the store_prices Kafka topic for downstream consumption.

2. Stream Processing (Lakehouse ETL Engine)

• Component: streaming_processor.ipynb
• Framework: Apache Spark Structured Streaming.
• Architecture Implementation:
  • Bronze Layer: Ingests raw historical data from Kafka in an append-only format.
  • Silver Layer: Performs data cleaning, parsing, and schema enforcement to ensure data quality.
  • Gold Layer: Executes complex windowed aggregations (20-second intervals) and computes technical indicators, including Volatility and Volume-Weighted Average Price (VWAP).

3. Machine Learning Integration (Real-Time Inference)

• Component: model.ipynb
• Model Architecture: Random Forest Classifier (Spark MLlib).
• Objective: Binary classification of short-term price direction (Up/Down).
• Integration: The pre-trained model is loaded directly into the streaming pipeline. It generates inference predictions and confidence probabilities for every micro-batch processed in the Gold layer.

4. Visualization Layer (FinanceLake Terminal)

• Component: dashboard.py
• Technology Stack: Streamlit & Plotly.
• Features:
  • Live Ticker Tape: Displays real-time price updates and trend indicators.
  • Technical Analysis: Visualizes key indicators including MACD, Bollinger Bands, and Stochastic Oscillators.
  • Predictive Analytics: Renders the AI model's confidence scores dynamically.
  • Data Access: Connects directly to Gold Delta tables to ensure data freshness and minimize latency.

Documentation & Setup

Note: The included README.md contains all necessary steps to run this pipeline from scratch.

It provides a comprehensive guide covering:

1. Infrastructure: Setting up Kafka/Zookeeper (Docker or Local).
2. Initialization: Generating the initial dataset and training the AI model (model.ipynb).
3. Execution: Interactive step-by-step instructions for the streaming processor (streaming_processor.ipynb).
4. Visualization: Launching the Streamlit terminal.

---

Implementation Verification

The following components have been verified and tested:

• Kafka Ingestion: Producer successfully streams live trade data to the broker.
• ETL Pipeline: Spark jobs correctly write to Bronze, Silver, and Gold Delta tables.
• Model Inference: Random Forest model performs live classification on streaming data.
• Dashboard Rendering: Streamlit application visualizes metrics and updates in real-time.
• Configuration Management: Environment variables are isolated and managed via .env.

---

Contributors

This feature was implemented by our team:

• MISDAK Abdessamad
• ABOUJAAFAR Mousaab
• AIT NACER Abdellah

---

Visual Demonstration

Live Pipeline Execution

This section showcases the end-to-end operation of the FinanceLake streaming pipeline. It demonstrates how real-time market data flows from ingestion to processing, model inference, and finally to the interactive dashboard, highlighting the system's low-latency analytics capabilities

🎥 Watch the demo video at this link: https://youtu.be/o1dPtFk3jgA

Terminal Interface

The FinanceLake Terminal provides an interactive visualization of live market data and predictive analytics. It displays real-time price movements, technical indicators, and AI model predictions, allowing users to monitor trends and insights with minimal latency.

---

Related Issues

Closes #37