Integration of Real-Time Stock Prediction Pipeline with Spark & GRU

[hafsaAit]

PULL REQUEST: Real-Time Stock Prediction Pipeline with Spark & GRU

🎯 CONTRIBUTION OBJECTIVE

Implement a complete real-time stock prediction pipeline combining:

• Apache Kafka (data ingestion)
• Apache Spark Streaming (processing)
• GRU - Gated Recurrent Unit (Deep Learning)
• WebSocket Dashboard (visualization)

🚀 MAIN FILES

01_train_gru.py

• GRU model training on synthetic data
• Generation of realistic time series
• Model (.h5) and scaler (.pkl) saving

02_kafka_producer.py

• Real-time Kafka producer (1 msg/sec)
• Simulation of 5 stock symbols
• Realistic patterns with timestamping

03_spark_gru_streaming.py

• Spark Streaming pipeline with GRU UDF
• Real-time buy/sell predictions
• Automatic BUY/SELL alerts

04_dashboard.py

• Flask dashboard
• Auto-refreshing real-time visualization
• Predictions and statistics display

🔧 TECHNICAL DEPENDENCIES

pyspark==3.5.0          # Spark Structured Streaming
kafka-python==2.0.2     # Kafka client
tensorflow==2.13.0      # Deep Learning GRU
flask-socketio==5.3.4   # WebSocket Dashboard

🎥 DEMONSTRATION

3-minute video demonstrating:

• Complete pipeline launch
• Real-time dashboard in action
• GRU predictions and alerts
• System architecture

📽️ Video link:
https://drive.google.com/file/d/12lGiE5UShwwOsJmRmTlS0Qumi0PnDvmy/view?usp=sharing

📁 STRUCTURE

spark-dl/
└── projet-spark-dl/
    ├── 01_train_gru.py
    ├── 02_kafka_producer.py
    ├── 03_spark_gru_streaming.py
    ├── 04_dashboard.py
    └── models/
        ├── gru_model.h5
        └── scaler.pkl

👥 CONTRIBUTORS

• Amina ESSAFI
• Hafsa AIT EL ATIK
• Chaimaa ELQOUBAA