Stock Price Prediction — MLOps Project

End-to-end MLOps pipeline that predicts next-day stock price direction (UP / DOWN) using a RandomForest classifier, MLflow, FastAPI, Prometheus, and Grafana.

---

Table of Contents

1. Project Overview
2. Directory Structure
3. Prerequisites & Installation
4. Configuration
5. Part A — Data Ingestion & Validation
6. Part B — Feature Engineering
7. Part C — Model Training & MLflow
8. Part D — FastAPI Serving
9. Part E — Monitoring, Drift & Retraining
10. Frontend — Flask Prediction App
11. Prometheus & Grafana
12. Docker Deployment
13. Custom Pipeline Scheduler
14. Running Tests
15. Port Reference & Troubleshooting

---

1. Project Overview

Item	Detail
Task	Binary classification: will next-day closing price be higher than today's?
Model	RandomForestClassifier (scikit-learn)
ML metric	F1-score, ROC-AUC
Business metric	Inference latency < 200 ms
Data source	Yahoo Finance via yfinance
Experiment tracking	MLflow (SQLite backend)
Data versioning	DVC
API	FastAPI (port 8000)
Frontend	Flask (port 5050)
Monitoring	Prometheus + Grafana
Containerisation	Docker + Docker Compose
Pipeline scheduling	Custom Python scheduler (scripts/run_pipeline_scheduler.py)

Documentation

Full project documentation is in the docs/ folder:

• Architecture Document
• High-Level Design
• Low-Level Design
• Test Plan & Cases
• User Manual

---

2. Directory Structure

Stock-Price-Prediction-Project/
├── configs/config.yaml              # Single source of truth for all parameters
├── data/
│   ├── raw/                         # Downloaded OHLCV CSVs (DVC tracked)
│   ├── interim/                     # Cleaned data
│   ├── processed/                   # Feature matrix (model input)
│   └── predictions/prediction_log.jsonl  # Every API request logged here
├── src/
│   ├── api/         main.py, routes.py, schemas.py, utils.py
│   ├── data/        ingest.py, validate.py, schema.yaml
│   ├── features/    build_features.py, indicators.py
│   ├── models/      train.py, evaluate.py, predict.py, registry.py
│   ├── monitoring/  metrics.py, drift.py, alerting.py
│   ├── pipelines/   data_pipeline.py, training_pipeline.py, retrain_pipeline.py
│   └── utils/       config_loader.py, logger.py, helpers.py
├── frontend/predict_app.py          # Simple Flask prediction UI
├── tests/           test_data.py, test_features.py, test_model.py, test_api.py
├── docker/          Dockerfiles, prometheus.yml, alert_rules.yml, grafana/
├── scripts/         Pipeline runners, scheduler, monitoring scripts
├── dvc.yaml                         # 5-stage DVC pipeline
├── mlflow.db                        # MLflow SQLite backend
├── conftest.py                      # Pytest path fix
└── setup.py                         # pip install -e .

---

3. Prerequisites & Installation

Requirements

• Python 3.11 (conda environment mlops_py311)
• Git, Docker (optional)

Install

git clone https://github.com/<your-username>/Stock-Price-Prediction-Project.git
cd Stock-Price-Prediction-Project

conda activate mlops_py311
pip install -r requirements.txt
pip install -e .          # makes src/ importable everywhere
chmod +x scripts/*.sh

Verify

python -c "from src.utils.config_loader import load_config; print('OK')"
pytest tests/ -q

---

4. Configuration

All parameters live in configs/config.yaml — nothing is hardcoded.

data:
  ticker:     "AAPL"
  start_date: "2018-01-01"
  end_date:   "2024-12-31"
  train_split: 0.70
  val_split:   0.15

model:
  params:
    n_estimators: 100
    max_depth:    10

mlflow:
  tracking_uri: "sqlite:///mlflow.db"
  artifact_root: "./mlruns"
  experiment_name: "stock-price-prediction"

monitoring:
  drift_threshold:            0.1
  alert_error_rate_threshold: 0.05

---

5. Part A — Data Ingestion & Validation

# Full data pipeline
./scripts/run_data_pipeline.sh

# Individual steps
python -m src.data.ingest
python -m src.data.validate --input data/raw/AAPL_raw_*.csv

# Via DVC
dvc repro

Outputs: data/raw/AAPL_raw_*.csv, artifacts/data_validation_report.json

---

6. Part B — Feature Engineering

Computes 22 features: log return, SMA (5, 20), RSI, MACD, Bollinger Bands, ATR, volatility, volume ratio, and lag returns (1, 2, 3, 5 days). Also saves drift baseline to artifacts/drift_baseline/stats.json.

# Runs automatically inside run_data_pipeline.sh
python -m src.features.build_features --input data/interim/AAPL_*_clean.csv

---

7. Part C — Model Training & MLflow

# Train + evaluate + promote to Production
./scripts/run_training.sh --auto-promote

# View experiments
mlflow ui --port 5000

Registry operations

python -m src.models.registry --list
python -m src.models.registry --promote --run-id <id> --stage Production
python -m src.models.registry --rollback

---

8. Part D — FastAPI Serving

uvicorn src.api.main:app --host 0.0.0.0 --port 8000 --reload

Endpoint	Description
GET /health	Liveness probe
GET /ready	Readiness — 200 only when model is loaded
POST /predict	Inference on 22-feature vector
GET /docs	Swagger UI

curl http://localhost:8000/health
curl http://localhost:8000/ready
curl -X POST http://localhost:8000/predict \
  -H "Content-Type: application/json" \
  -d '{"log_return":0.012,"pct_return":0.012,"sma_5":182.5,...}'

---

10. Frontend — Flask Prediction App

# Start FastAPI first, then:
python frontend/predict_app.py
# Open: http://localhost:5050

Enter a ticker symbol (e.g. AAPL), click Predict. The app fetches today's market data, computes all 22 features automatically, calls POST /predict, and displays UP / DOWN with confidence percentage.

---

11. Prometheus & Grafana

Local (no Docker)

# Terminal 1 — metrics exporter
python -m src.monitoring.metrics --port 8001

# Terminal 2 — Prometheus
./scripts/start_prometheus_local.sh --port 9090 --metrics-port 8001

# Terminal 3 — Grafana
./scripts/install_grafana.sh --port 3000 --prometheus-port 9090

Manual checks

python -m src.monitoring.drift
python -m src.monitoring.alerting

---

12. Docker Deployment

# Build and start all 4 services
docker-compose up --build

# Services:
# api         → http://localhost:8000
# monitoring  → http://localhost:8001/metrics
# prometheus  → http://localhost:9090
# grafana     → http://localhost:3000  (admin/admin)

---

13. Custom Pipeline Scheduler

# Run pipeline once
python scripts/run_pipeline_scheduler.py --run-once

# Run daily (every 24 hours)
python scripts/run_pipeline_scheduler.py

# Run every 6 hours
python scripts/run_pipeline_scheduler.py --interval-hours 6

---

14. Running Tests

pytest tests/ -v           # all 63 tests
pytest tests/test_data.py  # 12 tests
pytest tests/test_api.py   # 18 tests

No PYTHONPATH=. needed — conftest.py handles it.

---

15. Port Reference & Troubleshooting

Service	Default Port	Check
FastAPI	8000	curl http://localhost:8000/health
Flask UI	5050	http://localhost:5050
Metrics exporter	8001	curl http://localhost:8001/metrics
Prometheus	9090	http://localhost:9090/targets
Grafana	3000	http://localhost:3000
MLflow UI	5000	mlflow ui --port 5000

ModuleNotFoundError: No module named 'src'

pip install -e .

API returns 503 on /predict

python -m src.models.registry --list
./scripts/run_training.sh --auto-promote

Drift shows 0 features checked

python scripts/simulate_predictions.py --n 300 --mode drift --label
python -m src.monitoring.drift