Skip to content

Commit cf45a25

Browse files
hoangsonwwhoangsonww
authored
Merge pull request #272 from hoangsonww/feat/enhance-ci
Enhance implementation of MovieVerse AI components and configurations & sync code with private codebase
2 parents 47b4084 + bdc9e77 commit cf45a25

1,087 files changed

Lines changed: 60770 additions & 46815 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/README.md‎

Lines changed: 329 additions & 197 deletions
Large diffs are not rendered by default.

‎.idea/sqldialects.xml‎

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎DEPLOYMENT.md‎

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -176,17 +176,27 @@ flowchart LR
176176

177177
### Option 2: Kubernetes (EKS/GKE/AKS)
178178

179-
1. **Apply base resources:**
179+
Use the consolidated Kubernetes stack in `kubernetes/` for both the core microservices and rollout patterns.
180+
181+
1. **Apply microservices base resources:**
182+
```bash
183+
kubectl apply -f kubernetes/base/
184+
kubectl apply -f kubernetes/infra/
185+
kubectl apply -f kubernetes/services/
186+
kubectl apply -f kubernetes/edge/
187+
```
188+
189+
2. **Apply legacy blue-green base resources (optional):**
180190
```bash
181191
kubectl apply -f kubernetes/base/
182192
```
183193

184-
2. **Deploy using Blue-Green strategy:**
194+
3. **Deploy using Blue-Green strategy:**
185195
```bash
186196
kubectl apply -f kubernetes/blue-green/
187197
```
188198

189-
3. **Or deploy using Canary strategy:**
199+
4. **Or deploy using Canary strategy:**
190200
```bash
191201
kubectl apply -f kubernetes/canary/
192202
```

‎MovieVerse-AI/Dockerfile‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
FROM python:3.11-slim
2+
3+
WORKDIR /opt/movieverse-ai
4+
5+
ENV PYTHONDONTWRITEBYTECODE=1 \
6+
PYTHONUNBUFFERED=1
7+
8+
RUN apt-get update && apt-get install -y --no-install-recommends \
9+
build-essential \
10+
&& rm -rf /var/lib/apt/lists/*
11+
12+
COPY MovieVerse-AI/requirements.txt ./requirements.txt
13+
RUN pip install --no-cache-dir -r requirements.txt
14+
15+
COPY MovieVerse-AI/movieverse_ai ./movieverse_ai
16+
COPY MovieVerse-AI/feature_repo ./feature_repo
17+
18+
RUN mkdir -p /opt/movieverse-ai/models /opt/movieverse-ai/index
19+
20+
EXPOSE 9000
21+
22+
CMD ["uvicorn", "movieverse_ai.api.main:app", "--host", "0.0.0.0", "--port", "9000"]

‎MovieVerse-AI/Dockerfile.airflow‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
FROM apache/airflow:2.8.3
2+
3+
USER root
4+
RUN apt-get update && apt-get install -y --no-install-recommends \
5+
build-essential \
6+
&& rm -rf /var/lib/apt/lists/*
7+
USER airflow
8+
9+
COPY MovieVerse-AI/requirements.txt /tmp/requirements.txt
10+
RUN pip install --no-cache-dir -r /tmp/requirements.txt
11+
12+
COPY MovieVerse-AI/movieverse_ai /opt/airflow/movieverse_ai
13+
COPY MovieVerse-AI/feature_repo /opt/airflow/feature_repo
14+
15+
ENV PYTHONPATH=/opt/airflow

‎MovieVerse-AI/README.md‎

Lines changed: 183 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,183 @@
1+
# MovieVerse AI Platform
2+
3+
MovieVerse-AI is a production-grade machine learning platform that powers recommendations, similarity search, sentiment analysis, and ranking for MovieVerse. It includes data ingestion, feature store, model training, registry, inference APIs, and monitoring.
4+
5+
## Capabilities
6+
7+
- Personalized recommendations (collaborative + content-aware)
8+
- Movie similarity search using vector embeddings
9+
- Review sentiment classification
10+
- Ranking model for feed/search ordering
11+
- Feature store with offline/online serving (Feast + Redis)
12+
- MLflow model registry and experiment tracking
13+
- Kafka-driven event ingestion
14+
- Drift monitoring with Evidently
15+
16+
## Stack
17+
18+
- Python 3.11
19+
- FastAPI + Uvicorn for online inference
20+
- LightFM for recommender models
21+
- SentenceTransformers + FAISS for embeddings
22+
- scikit-learn for sentiment and ranking models
23+
- Feast for feature storage
24+
- MLflow for registry and experiments
25+
- Kafka for event streaming
26+
- PostgreSQL + MySQL + MongoDB data sources
27+
- Redis online store
28+
29+
## Architecture
30+
31+
Online inference flow:
32+
33+
```mermaid
34+
flowchart LR
35+
Backend[MovieVerse Backend] -->|HTTP| API[MovieVerse AI API]
36+
API --> Reco[Recommender Service]
37+
API --> Similar[Similarity + Embeddings]
38+
API --> Ranker[Ranking Service]
39+
API --> Sentiment[Sentiment Service]
40+
API --> Summarize[Summarization Service]
41+
API --> Vision[Vision Service]
42+
Reco --> Redis[(Redis Online Store)]
43+
Similar --> FAISS[(FAISS Index)]
44+
API --> Feast[Feature Store]
45+
Feast --> Redis
46+
Feast --> Postgres[(PostgreSQL Feature Repo)]
47+
API --> MLflow[(MLflow Registry)]
48+
```
49+
50+
Training + feature pipeline flow:
51+
52+
```mermaid
53+
flowchart TD
54+
Kafka[(Kafka Events)] --> Ingest[Event Ingestion]
55+
MySQL[(MySQL Movies)] --> Ingest
56+
Postgres[(PostgreSQL Ratings/Reviews)] --> Ingest
57+
Mongo[(Mongo Metadata)] --> Ingest
58+
Ingest --> Features[Build Feature Tables]
59+
Features --> Feast[Feast Feature Store]
60+
Feast --> Materialize[Materialize Online Features]
61+
Materialize --> Redis[(Redis Online Store)]
62+
Features --> Train[Train Models]
63+
Train --> MLflow[(MLflow Registry)]
64+
MLflow --> Sync[Sync Latest Models]
65+
Sync --> API[MovieVerse AI API]
66+
```
67+
68+
## Directory Layout
69+
70+
```
71+
MovieVerse-AI/
72+
├── movieverse_ai/ # Core package (API, models, pipelines)
73+
├── feature_repo/ # Feast feature store definitions
74+
├── airflow/dags/ # Training + materialization workflows
75+
├── sql/ # Database schema for AI data
76+
├── k8s/ # Kubernetes deployment assets
77+
├── docker-compose.ai.yml # Local AI stack runtime
78+
├── docker-compose.airflow.yml # Airflow scheduler/runtime
79+
├── requirements.txt # Production dependencies
80+
└── Dockerfile # Inference service image
81+
```
82+
83+
## Services
84+
85+
- `movieverse_ai.api.main`: Online inference API
86+
- `movieverse_ai.pipelines.ingest_events`: Kafka ingestion into Postgres
87+
- `movieverse_ai.pipelines.train_recommender`: Recommendation training
88+
- `movieverse_ai.pipelines.train_sentiment`: Review sentiment training
89+
- `movieverse_ai.pipelines.build_ranking_features`: Ranking feature generation
90+
- `movieverse_ai.pipelines.train_ranker`: Ranking model training
91+
- `movieverse_ai.pipelines.build_embeddings`: Embedding + FAISS index
92+
- `movieverse_ai.pipelines.build_feature_tables`: User/movie feature tables for Feast
93+
- `movieverse_ai.pipelines.materialize_features`: Feast materialization
94+
- `movieverse_ai.pipelines.sync_models`: Pull latest artifacts from MLflow
95+
- `movieverse_ai.monitoring.drift_monitor`: Drift reports
96+
97+
## Local Run
98+
99+
1) Start the AI stack (Postgres, Redis, Kafka, MLflow, MinIO):
100+
101+
```bash
102+
docker compose -f MovieVerse-AI/docker-compose.ai.yml up -d
103+
```
104+
105+
The AI API is exposed at `http://localhost:9100` when running via Docker Compose.
106+
107+
2) Initialize the AI database schema:
108+
109+
```bash
110+
psql postgresql://movieverse:movieverse@localhost:5433/movieverse_ai \
111+
-f MovieVerse-AI/sql/postgres_init.sql
112+
```
113+
114+
3) (Optional) Start Airflow for scheduled pipelines:
115+
116+
```bash
117+
psql postgresql://movieverse:movieverse@localhost:5433/postgres -c "CREATE DATABASE airflow"
118+
docker compose -f MovieVerse-AI/docker-compose.airflow.yml up -d
119+
```
120+
121+
Airflow connects to the AI stack over the `movieverse-ai` Docker network, so start the AI stack first.
122+
123+
4) Train models and build embeddings:
124+
125+
```bash
126+
cd MovieVerse-AI
127+
python -m movieverse_ai.pipelines.train_recommender
128+
python -m movieverse_ai.pipelines.train_sentiment
129+
python -m movieverse_ai.pipelines.build_ranking_features
130+
python -m movieverse_ai.pipelines.train_ranker
131+
python -m movieverse_ai.pipelines.build_embeddings
132+
python -m movieverse_ai.pipelines.build_feature_tables
133+
```
134+
135+
5) Start the inference API:
136+
137+
```bash
138+
cd MovieVerse-AI
139+
uvicorn movieverse_ai.api.main:app --host 0.0.0.0 --port 9000
140+
```
141+
142+
## Environment Variables
143+
144+
All configuration is driven by `MOVIEVERSE_AI_` prefixed environment variables. Defaults are set for the local AI stack.
145+
146+
Key variables:
147+
148+
- `MOVIEVERSE_AI_POSTGRES_DSN`
149+
- `MOVIEVERSE_AI_MYSQL_DSN`
150+
- `MOVIEVERSE_AI_MONGO_URI`
151+
- `MOVIEVERSE_AI_REDIS_URL`
152+
- `MOVIEVERSE_AI_KAFKA_BOOTSTRAP_SERVERS`
153+
- `MOVIEVERSE_AI_MLFLOW_TRACKING_URI`
154+
- `MOVIEVERSE_AI_S3_ENDPOINT_URL`
155+
156+
## Production Deployment
157+
158+
- Kubernetes manifests live in `MovieVerse-AI/k8s/` and deploy the inference API.
159+
- Use managed AWS services (RDS, MSK, OpenSearch, ElastiCache) and point the AI stack to them via environment variables.
160+
- MLflow artifacts are stored in S3/MinIO; align this with the same bucket used in your infra stack.
161+
162+
## Data Requirements
163+
164+
The AI pipelines expect these tables:
165+
166+
- `ratings` (user_id, movie_id, rating, created_at)
167+
- `reviews` (user_id, movie_id, rating, review_text, created_at)
168+
- `ranking_features` (movie_id, popularity, avg_rating, rating_count, recency_days, label)
169+
- `user_features` and `movie_features` for Feast
170+
171+
Movie metadata is sourced from MySQL `movies` (movie_id, title, overview, genres, release_date).
172+
173+
## API Endpoints
174+
175+
- `GET /healthz`
176+
- `GET /metrics`
177+
- `POST /recommendations`
178+
- `POST /similar`
179+
- `POST /sentiment`
180+
- `POST /rank`
181+
- `POST /summarize`
182+
- `POST /genres/classify`
183+
- `POST /vision/labels`

‎MovieVerse-AI/airflow/dags/movieverse_ai_pipeline.py‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
from datetime import datetime, timedelta
2+
3+
from airflow import DAG
4+
from airflow.operators.bash import BashOperator
5+
6+
DEFAULT_ARGS = {
7+
"owner": "movieverse-ai",
8+
"depends_on_past": False,
9+
"retries": 1,
10+
"retry_delay": timedelta(minutes=10),
11+
}
12+
13+
with DAG(
14+
dag_id="movieverse_ai_pipeline",
15+
default_args=DEFAULT_ARGS,
16+
description="MovieVerse AI training and feature materialization",
17+
schedule_interval="0 2 * * *",
18+
start_date=datetime(2024, 1, 1),
19+
catchup=False,
20+
max_active_runs=1,
21+
) as dag:
22+
train_recommender = BashOperator(
23+
task_id="train_recommender",
24+
bash_command="python -m movieverse_ai.pipelines.train_recommender",
25+
)
26+
27+
train_sentiment = BashOperator(
28+
task_id="train_sentiment",
29+
bash_command="python -m movieverse_ai.pipelines.train_sentiment",
30+
)
31+
32+
train_ranker = BashOperator(
33+
task_id="train_ranker",
34+
bash_command="python -m movieverse_ai.pipelines.train_ranker",
35+
)
36+
37+
build_ranking_features = BashOperator(
38+
task_id="build_ranking_features",
39+
bash_command="python -m movieverse_ai.pipelines.build_ranking_features",
40+
)
41+
42+
build_embeddings = BashOperator(
43+
task_id="build_embeddings",
44+
bash_command="python -m movieverse_ai.pipelines.build_embeddings",
45+
)
46+
47+
materialize_features = BashOperator(
48+
task_id="materialize_features",
49+
bash_command="python -m movieverse_ai.pipelines.materialize_features",
50+
)
51+
52+
build_feature_tables = BashOperator(
53+
task_id="build_feature_tables",
54+
bash_command="python -m movieverse_ai.pipelines.build_feature_tables",
55+
)
56+
57+
(train_recommender, train_sentiment) >> build_embeddings
58+
build_ranking_features >> train_ranker
59+
(build_embeddings, train_ranker) >> build_feature_tables >> materialize_features

0 commit comments

Comments
 (0)