Mini Projet

[roukhmi]

FinanceLake

What is FinanceLake?

FinanceLake is an open-source financial data platform that ingests, analyzes, and visualizes market and financial data — similar in ambition to platforms like Bloomberg Terminal, but powered by open technologies.

Whether you're a quant, data engineer, open-source maintainer, or trading enthusiast, FinanceLake offers a scalable and intelligent data stack to support real-time insights, financial research, and data-driven decision-making.

---

🚀 Features

• 📥 Data Ingestion
  Real-time streaming pipelines using Apache Kafka with Delta Lake Bronze layer storage and batch processing via Apache NiFi.

• 🏗️ Delta Lake Architecture
  Multi-layer data lakehouse with Bronze (raw), Silver (cleaned), and Gold (analytics) layers for ACID transactions and time travel.

• 🤖 Machine Learning Pipeline
  Real-time ML model training with Spark MLlib Random Forest classifier for price direction prediction and continuous model updates.

• ⚙️ Big Data Processing
  Built on Apache Spark 3.4.1 with Delta Lake 2.4.0 for scalable, resilient analytics and schema evolution.

• 📈 Advanced Financial Analytics
  Technical indicators (RSI, MACD), volatility analysis, risk metrics, and custom financial KPIs with real-time computation.

• 📊 Interactive Real-time Dashboard
  Professional Streamlit dashboard with live data streaming, technical analysis charts, ML insights, and risk analytics.

• 🧠 Query Engine
  Ask questions and get answers using a simple SQL-like interface or a natural language layer (NLQ) with optional LLM integration.

• 📡 Data APIs
  REST & GraphQL APIs to expose insights and dashboards to downstream systems or external apps.

---

🏗️ Architecture Overview

FinanceLake implements a modern Delta Lakehouse Architecture with real-time streaming and ML capabilities:

graph TB
    A[📡 Data Sources] --> B[Kafka Producer]
    B --> C[Kafka Broker]
    C --> D[🟡 Bronze Layer<br/>Raw Data]
    D --> E[Spark Streaming]
    E --> F[🔵 Silver Layer<br/>Cleaned + ML]
    F --> G[Spark ML<br/>Model Training]
    F --> H[Spark SQL<br/>Aggregations]
    G --> I[🟢 Gold Layer<br/>KPIs & Predictions]
    H --> I
    I --> J[📊 Streamlit Dashboard]

    style D fill:#f59e0b
    style F fill:#3b82f6
    style I fill:#10b981
    style J fill:#8b5cf6

Loading

Data Pipeline Flow

1. Bronze Layer: Raw financial data ingestion from Kafka streams with Delta Lake ACID guarantees
2. Silver Layer: Data cleaning, validation, technical indicators calculation, and ML model training
3. Gold Layer: Aggregated KPIs, risk metrics, and ML predictions for analytics and reporting

Key Components

• Finance Data Generator: Simulates real-time financial data with technical indicators
• Spark Delta Processor: Streaming pipeline with ML training every 5 batches
• Streamlit Dashboard: Real-time visualization with professional UI
• Delta Lake Storage: ACID-compliant lakehouse with time travel capabilities

---

💡 Use Cases

• Market trend monitoring for trading teams
• Quantitative research and strategy testing
• Portfolio performance visualization
• Risk metrics computation
• Real-time financial data streaming and alerting

🎯 What can be accomplished with FinanceLake?

FinanceLake empowers users to unlock value from vast streams of financial and economic data. Here’s what you can achieve:

🧠 Derive Actionable Insights

• Track price movements, volatility, and trends across global markets
• Identify leading/lagging indicators to guide investment decisions
• Measure performance against custom benchmarks or indices

📈 Build & Test Trading Strategies

• Backtest strategies using historical tick/ohlcv data
• Generate buy/sell signals using technical indicators (RSI, MACD, EMA…)
• Evaluate drawdown, Sharpe ratio, beta, and other risk metrics

📊 Visualize and Monitor in Real Time

• Build dynamic dashboards to monitor positions, portfolios, and KPIs
• Stream live feeds for asset prices, news sentiment, or macro indicators
• Trigger alerts on thresholds or anomalies (via webhook, email, Slack)

🔎 Query Like a Pro

• Explore structured and unstructured financial data using SQL or natural language
• Query fundamentals, earnings, economic events, ESG scores, and more
• Slice and dice data per sector, geography, or custom segments

🏗️ Build Custom Financial Applications

• Create custom dashboards for hedge funds, fintech apps, or research teams
• Feed data into machine learning pipelines (e.g., predictive models)
• Connect external systems (trading bots, ML models, BI tools) via API

🧩 Extend and Contribute

• Add custom connectors to new data sources (e.g., crypto exchanges, alt-data)
• Contribute notebooks, indicators, or data visualizations
• Help shape the roadmap of an open, transparent financial platfor

👉 Live Demos

Comming soon !!

💪 Supported Data Sources

Comming soon !!

🚀 Getting Started

Prerequisites

• Docker & Docker Compose
• Git
• At least 8GB RAM (recommended 16GB)
• 10GB+ free disk space

Quick Start with Docker Compose

1. Clone the repository

   git clone https://github.com/roukhmi/financelake.git
   cd financelake

2. Start the Delta Lake Platform

   docker-compose up --build

3. Access the Applications

   • Real-time Dashboard: http://localhost:8501
   • Apache Superset: http://localhost:8088 (admin/admin)
   • Kafka: localhost:9092 (internal)

Manual Development Setup

# 1. Install dependencies
pip install -r requirements.txt

# 2. Start Kafka (in separate terminals)
# Terminal 1: Start Zookeeper
bin/zookeeper-server-start.sh config/zookeeper.properties

# Terminal 2: Start Kafka
bin/kafka-server-start.sh config/server.properties

# 3. Start the components
python finance_data_generator.py     # Terminal 3
python spark_delta_processor.py      # Terminal 4
streamlit run finance_dashboard.py   # Terminal 5

Configuration

Create a .env file in the project root:

# Kafka Configuration
KAFKA_BOOTSTRAP_SERVERS=kafka:9092

# Storage Paths
STORAGE_PATH=/app/storage

# Spark Configuration
SPARK_MASTER=local[*]

# Optional: Enable real Yahoo Finance data
USE_REAL_DATA=false

🎥 Project Demo Video

A demonstration video of the FinanceLake platform is available in this repository:

📹 Watch Demo Video

File: videos/financelake-demo.mp4
Format: MP4
Duration: ~5-10 minutes

How to View the Video

1. Via GitHub: Click the video link above (GitHub will play MP4 files directly)
2. Locally: After cloning the repository, open videos/financelake-demo.mp4 with any video player
3. VS Code: Right-click on the video file and select "Open Preview"

Video Contents

This video demonstrates:

• Real-time data ingestion pipeline
• Delta Lake architecture (Bronze → Silver → Gold)
• Interactive Streamlit dashboard
• Machine learning price predictions
• Multi-asset analytics capabilities
• Docker deployment process

---

🤓 Usage

Dashboard Features

The FinanceLake dashboard provides comprehensive real-time financial analytics:

📊 Real-time Analytics

• Live stock price charts with technical indicators
• RSI (Relative Strength Index) and MACD signals
• Volume analysis and price change tracking
• Auto-refresh every 30 seconds

📈 Multi-asset Support

• AAPL, GOOGL, MSFT, AMZN, TSLA, NVDA, META, NFLX
• Major indices: S&P 500, Dow Jones, NASDAQ
• Commodities: Gold, Oil
• Forex: EUR/USD

🔄 Pipeline Monitoring

• Real-time pipeline health status
• Data flow visualization (Bronze → Silver → Gold)
• Record counts and processing timestamps

🤖 ML Insights

• Model accuracy gauge (Random Forest: ~72%)
• Feature importance analysis (RSI, MACD, Volatility)
• Prediction confidence metrics

💹 Risk Analytics

• Volatility analysis and risk indicators
• Overbought/oversold percentages
• Bullish percentage calculations

Data Schema

{
  "timestamp": "2024-01-10T12:00:00Z",
  "symbol": "AAPL",
  "price": 150.25,
  "open": 149.50,
  "high": 151.00,
  "low": 148.75,
  "close": 150.25,
  "volume": 15420,
  "rsi": 65.4,
  "macd": 0.85,
  "volatility": 2.1,
  "price_change": 1.25,
  "price_direction": 1,
  "volume_change": 234
}

Contributing

Please read the contribution guidelines before you make contribution. The following docs list the resources you might need to know after you decided to make contribution.

• Create an Issue: Report a bug or feature request to FinanceLake
• Submit a PR: Start with good first issues or issues with no assignees
• Join Mailing list: Initiate or participate in project discussions on the mailing list
• Write a Blog: Write a blog to share your use cases about FinanceLake
• Develop a Plugin: Integrate FinanceLake with more data sources as requested by the community

👩🏾‍💻 Contributing Code

If you plan to contribute code to FinanceLake, we have instructions on how to get started with setting up your Development environemtn.

• Developer Setup Instructions
• Development Workflow

📄 Contributing Documentation

One of the best ways to get started contributing is by improving FinanceLake's documentation.

• FinanceLake's documentation is hosted at FinanceLake
• We have a separate GitHub repository for FinanceLack's documentation: github.com/FinanceLake/financelake-docs

⌚ Roadmap

• Roadmap: Detailed roadmaps for FinanceLake.

💙 Community

Message us on Discord

📄 License

🔧 Environment Configuration

Before running the project, configure your environment variables.

1. Copy the .env.example file and create your own .env file:

cp .env.example .env

2. Edit the .env file and fill in your specific configuration:

-DB_HOST: Database host (e.g., localhost)
-KAFKA_BROKER: Kafka broker address
-SPARK_MASTER Spark master URL
-API_KEY: Your API key
-DATA_SOURCE_URL: URL to fetch data from
-RAW_DATA_PATH: Path for storing raw data
-DASHBOARD_USER: Dashboard login user
-LOG_LEVEL: Logging level (e.g., INFO, DEBUG)

Steps to execute Kafka & HDFS:

1. Start Kafka & Zookeeper

bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties

2. Start HDFS

Make sure HDFS is running on localhost:9000.

3. Start the Kafka producer

python producer.py

4. Deploy the HDFS connector

curl -X POST -H "Content-Type: application/json" --data @hdfs-sink.json http://localhost:8083/connectors

---

🤝 Contributors

This project was built as a collaborative school project by:

• [Anas Roukhmi]
• [Yassine Bouyahia]
• [Abdelkrim Rekbi]