Complete FinanceLake fix: Spark 3.5, Delta 3.0, and Streamlit Dashboa…

[cldanass]

📊 FinanceLake – Real-Time Financial Data Platform

FinanceLake is an end-to-end data engineering and machine learning pipeline designed for real-time and batch financial data processing. It follows the industry-standard Medallion Architecture to ensure data quality, scalability, and reliability, while enabling analytics and live price prediction.

---

🏗️ System Architecture

The project is built around the Medallion Architecture, a proven pattern in modern data platforms:

🥉 Bronze Layer (Raw)

• Ingests raw JSON data from the Alpha Vantage API
• Stores data in its original form
• Allows full reprocessing and auditability

🥈 Silver Layer (Cleaned)

• Schema enforcement
• Data typing (dates, floats, etc.)
• Deduplication using Apache Spark
• Produces clean and reliable datasets

🥇 Gold Layer (Aggregated)

• Business-level aggregates via Spark SQL
• Includes:
  • Daily stock metrics
  • Market-wide summaries
• Optimized for analytics and dashboards

🤖 Machine Learning

• Linear Regression model trained on Silver data
• Used during the streaming phase for:
  • Real-time price prediction
  • Live analytics integration

---

🛠️ Tech Stack

Category	Technology
Processing	Apache Spark 3.5.0
Storage	Delta Lake 3.0.0 (ACID, Time Travel)
Language	Python 3.12
Database Interface	Spark SQL & Delta-rs
Visualization	Streamlit (2026 Edition)

---

🚀 Features

• Batch ingestion of historical financial data
• Streaming simulation for real-time pipelines
• ACID-compliant storage with Delta Lake
• SQL-based analytics on Gold tables
• Machine learning for live price prediction
• Interactive dashboards with Streamlit

---

👥 Contributors

• ANAS EL HALLABI
• YASSMINE EL BOUKHARI

🎥 Video Demonstration

(https://drive.google.com/file/d/1gn6_PwcFFTvR4x5-oiBvTiikbhQteJgn/view?usp=sharing)

[pr-elhajji]

Thanks for the contribution! Looking through this, it seems some of the Figma requirements weren't fully met. Let’s sync up to make sure we’re on the same page, or feel free to take another pass at it.