A production-style recommendation system built on the MovieLens 25M dataset using the same architecture behind YouTube, Netflix, and Spotify recommendations.
Most people think recommendation = "find similar movies."
The real problem is: out of 57,000 movies, how do you find the right 10 in milliseconds?
The Two-Tower model solves this by converting everything into numbers (vectors) and finding which numbers are closest to each other.
┌─────────────────────────────────────────────────────────────────┐
│ THE CORE IDEA │
│ │
│ "Convert users and movies into the same language (vectors) │
│ and measure how close they are." │
└─────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────┐
│ TWO TOWER ARCHITECTURE │
└─────────────────────────────────────────┘
TOWER 1 (User) TOWER 2 (Item)
────────────── ──────────────
User History Movie Info
┌──────────┐ ┌──────────┐
│ Watched │ │ Genres │
│ Genres │ │ Year │
│ Ratings │ │ Tags │
│ Activity │ │ Mood │
└────┬─────┘ └────┬─────┘
│ │
┌────▼─────┐ ┌────▼─────┐
│ Linear │ │ Linear │
│ Layer 1 │ │ Layer 1 │
│ 223→256 │ │ 220→256 │
└────┬─────┘ └────┬─────┘
│ ReLU + BatchNorm │ ReLU + BatchNorm
┌────▼─────┐ ┌────▼─────┐
│ Linear │ │ Linear │
│ Layer 2 │ │ Layer 2 │
│ 256→128 │ │ 256→128 │
└────┬─────┘ └────┬─────┘
│ L2 Normalize │ L2 Normalize
│ │
▼ ▼
┌─────────┐ ┌─────────┐
│ User │ │ Item │
│ Vector │ │ Vector │
│ (128,) │ │ (128,) │
└────┬────┘ └────┬────┘
│ │
└──────────── DOT PRODUCT ───────────┘
│
▼
┌─────────┐
│ Score │ High = Good Match ✅
│ 0→1 │ Low = Bad Match ❌
└─────────┘
❌ NAIVE APPROACH (too slow):
For every request → compare user vs ALL 57,000 movies live
= 57,000 calculations per user per request
At 1M users = 57 BILLION calculations. Impossible.
✅ TWO TOWER APPROACH:
Step 1 (offline): Pre-compute all 57,000 movie vectors once
Store in FAISS index
Step 2 (online): Compute ONE user vector per request
FAISS finds top 10 in microseconds ⚡
┌─────────────────────────────────────────────────────────────────────┐
│ COMPLETE PIPELINE │
└─────────────────────────────────────────────────────────────────────┘
PHASE 1: DATA PHASE 2: TRAINING
───────────── ─────────────────
MovieLens 25M Positive pairs
┌───────────┐ (user liked movie)
│ratings.csv│──► preprocess ──► ┌─────────────┐
│movies.csv │ ───────────── │ train.csv │
│genome.csv │ │ val.csv │──► Two Tower ──► best_model.pt
└───────────┘ │ test.csv │ Training
└─────────────┘
item_features.py ▲
┌──────────────────┐ │
│ genres (19) │──────────────────────────┤
│ year (1) │ Item Tower input │
│ genome tags (100)│ 220 features │
└──────────────────┘ │
│
user_features.py │
┌──────────────────┐ │
│ taste profile │──────────────────────────┘
│ genre prefs (19) │ User Tower input
│ behavior stats │ 223 features
└──────────────────┘
PHASE 3: INDEXING PHASE 4: SERVING
───────────────── ────────────────
best_model.pt GET /recommend/42
│ │
▼ ▼
export_vectors.py ┌─────────────────┐
│ │ Load user vec │
▼ │ from .npy file │
item_vectors.npy └────────┬────────┘
(57k × 128) │
│ ▼
▼ ┌─────────────────┐
faiss_index.py │ Query FAISS │
│ │ index │
▼ └────────┬────────┘
movie_hnsw.index │
(fast lookup) ▼
Top 10 Movies 🎬
Problem: We only know what users LIKED. How do we teach "bad matches"?
Solution: In a batch of 256 (user, movie) pairs...
┌──────────────────────────────────────┐
│ User 0 → Movie 0 ← POSITIVE ✅ │
│ User 0 → Movie 1 ← NEGATIVE ❌ │ (we assume they didn't like it)
│ User 0 → Movie 2 ← NEGATIVE ❌ │
│ ... │
│ User 1 → Movie 0 ← NEGATIVE ❌ │
│ User 1 → Movie 1 ← POSITIVE ✅ │
└──────────────────────────────────────┘
256 positive pairs → 256×256 = 65,536 training signals per batch!
This is why it's so efficient.
Without FAISS: Compare user vs 57,000 movies = slow 🐢
With FAISS: Navigate a smart graph = microseconds ⚡
HNSW Graph (simplified):
Layer 2 (sparse): A ──────────── E ──────── H
Layer 1: A ── B ─── D ─ E ── F ── H
Layer 0 (dense): A─B─C─D─E─F─G─H─I─J─K...
Query enters at top → navigates down → finds nearest neighbors fast.
Recall vs exact search: >95% accurate.
Raw vector: [3.2, 1.1, 4.5, 0.8] ← magnitude varies
After L2 norm: [0.58, 0.20, 0.81, 0.14] ← always on unit sphere
Why? Dot product on unit sphere = cosine similarity (0 to 1)
Makes all scores comparable and training stable.
MovieLens 25M from GroupLens
25,000,095 ratings
162,541 users
57,361 movies
Time span: 1995 → 2019
Files used:
ratings.csv → userId, movieId, rating, timestamp
movies.csv → movieId, title, genres
genome-scores.csv→ movieId, tagId, relevance (semantic tags)
genome-tags.csv → tagId, tag name
movie_twotower/
│
├── data/
│ ├── raw/ ← ml-25m dataset
│ ├── processed/ ← cleaned features + train/val/test splits
│ ├── vectors/ ← exported embeddings (.npy files)
│ ├── checkpoints/ ← saved model weights
│ └── index/ ← FAISS index files
│
├── features/
│ ├── item_features.py ← builds movie feature matrix (220 dims)
│ └── user_features.py ← builds user feature matrix (223 dims)
│
├── models/
│ └── two_tower.py ← PyTorch model + loss function
│
├── pipeline/
│ ├── preprocess.py ← loads & splits ratings.csv
│ ├── train.py ← training loop
│ └── export_vectors.py ← pushes all movies through Item Tower
│
├── indexer/
│ └── faiss_index.py ← builds HNSW index + demo queries
│
├── api/
│ ├── main.py ← FastAPI server
│ └── schemas.py ← request/response models
│
├── config.yaml ← all hyperparameters
├── requirements.txt
├── Dockerfile
└── README.md
# 1. Clone the repo
git clone https://github.com/yourusername/movie_twotower.git
cd movie_twotower
# 2. Create virtual environment
python -m venv .venv
source .venv/bin/activate # Mac/Linux
.venv\Scripts\activate # Windows
# 3. Install dependencies
pip install -r requirements.txt
# 4. Download MovieLens 25M into ml-25m/ folder
# https://grouplens.org/datasets/movielens/25m/# Step 1 — Preprocess ratings
python pipeline/preprocess.py
# Step 2 — Build item features (movies → 220-dim vectors)
python features/item_features.py
# Step 3 — Build user features (users → 223-dim vectors)
python features/user_features.py
# Step 4 — Train the Two-Tower model
python pipeline/train.py
# Step 5 — Export embeddings for all movies + users
python pipeline/export_vectors.py
# Step 6 — Build FAISS HNSW index
python indexer/faiss_index.py
# Step 7 — Start the API
uvicorn api.main:app --reload# Health check
curl http://127.0.0.1:8000/health
# Get top-10 recommendations for user 42
curl http://127.0.0.1:8000/recommend/42Example response:
{
"user_idx": 42,
"recommendations": [
{"rank": 1, "movieId": 318, "title": "Shawshank Redemption (1994)", "score": 0.91},
{"rank": 2, "movieId": 296, "title": "Pulp Fiction (1994)", "score": 0.88},
{"rank": 3, "movieId": 2571, "title": "Matrix, The (1999)", "score": 0.85}
]
}# Build
docker build -t movie-twotower:latest .
# Run (mount data/ so container can access processed files)
docker run --rm -p 8000:8000 \
-v "$PWD/data":/app/data \
movie-twotower:latest| Component | Technology |
|---|---|
| Model | PyTorch |
| Vector Search | FAISS (HNSW) |
| API | FastAPI |
| Data Processing | Pandas, NumPy |
| Feature Scaling | Scikit-learn |
| Config | PyYAML |
| Hardware | Apple M2 MPS / CUDA / CPU |
OpenMP error on macOS:
export KMP_DUPLICATE_LIB_OK=TRUEFAISS not found:
pip install faiss-cpuMPS device error:
# Ensure PyTorch >= 2.0
pip install torch --upgrade✅ Two Tower architecture and why it scales
✅ Feature engineering for recommendation systems
✅ In-batch negative sampling
✅ PyTorch model building + training loops
✅ Recall@K evaluation metric
✅ FAISS approximate nearest neighbor search
✅ FastAPI serving
✅ Production ML pipeline structure
- MovieLens Dataset by GroupLens, University of Minnesota (CC BY)
- FAISS by Facebook AI Research
- Architecture inspired by YouTube Deep Neural Networks for YouTube Recommendations (Covington et al., 2016)
Built for learning. The same concepts power recommendations at YouTube, Netflix, and Spotify.