[EXTERNAL] docs(matrix-factorization): add measurable validation, outputs, and performance checks to subject and audit

subjects/ai/matrix-factorization/README.md

@@ -19,31 +19,69 @@ The goal of this project is to understand and apply advanced matrix factorizatio
 1. **Download the [MovieLens Dataset](https://grouplens.org/datasets/movielens/1m/)** (ratings, users, and movies).
 2. Preprocess the dataset to remove null values and prepare it for matrix factorization.
 3. Create a user-item interaction matrix from the data.
+4. Split the data into training and testing sets using a fixed `random_state = 42`.
+5. Normalize the user–item interaction matrix and save it under `processed/user_item_matrix.csv`.
 
 #### Singular Value Decomposition (SVD) Model
 
 1. Implement the SVD algorithm using the **scipy.sparse.linalg.svds** function for matrix factorization.
 2. Train the SVD model on the MovieLens dataset to generate predicted ratings for all users.
+3. Compute RMSE on the test set and append the value to `reports/model_metrics.json`.
+4. Save the full predicted rating matrix as `reports/svd_predictions.npy`.
 
 #### Probabilistic Matrix Factorization (PMF) Model
 
 1. Implement the PMF algorithm.
 2. Train the PMF model and visualize the model's convergence (e.g., plot Mean Squared Error over iterations).
+3. During training, log the Mean Squared Error (MSE) at each iteration/epoch.
+4. Generate and save a convergence plot (`MSE vs. iteration`) as `reports/pmf_convergence.png`.
+5. Save the learned latent factor matrices (`U` and `V`) under `reports/pmf_factors/`.
 
 #### Model Comparison and Evaluation
 
 1. Compare the performance of SVD and PMF using evaluation metrics such as **Mean Squared Error (MSE)**.
 2. Provide visual comparisons between the models using **matplotlib** to plot predicted vs. actual ratings.
+3. Save consolidated evaluation results as JSON:
+   `reports/model_metrics.json`
+
+   Example format:
+
+   ```json
+   {
+     "SVD_RMSE": 0.91,
+     "PMF_RMSE": 0.85,
+     "PMF_vs_SVD_improvement_%": 6.6
+   }
+   ```
+
+   - Generate and save comparison plots:
+     - Predicted vs Actual ratings: `reports/predicted_vs_actual.png`
+     - RMSE comparison (bar chart): `reports/rmse_comparison.png`
+   - Minimum expected performance:
+     - SVD RMSE ≤ 0.90
+     - PMF RMSE ≤ 0.85
+     - PMF improvement ≥ 5% over SVD
 
 #### Recommendation Generation
 
 1. Implement a function that generates movie recommendations for a user based on the predicted ratings from both the SVD and PMF models.
 2. Display top-rated movies for users and compare recommendations from both models.
+3. Implement in `utils/recommendation.py`:
+
+```python
+def generate_recommendations(user_id, model, top_n=10):
+    ...
+```
+
+4. Save the top-10 recommendations for each evaluated user in `reports/user_<id>_recommendations.csv`
 
 #### Analysis and Visualization
 
 1. Provide visualizations comparing SVD and PMF predictions for the same user.
 2. Offer insights into how the models differ in recommending movies for specific users based on their ratings history.
+3. Save the following plots under `reports/`:
+   - `user_comparison.png` — SVD vs PMF predictions for a selected user
+   - `top_recommendations.png` — Histogram (or bar chart) of top recommended movies
 
 #### Streamlit Dashboard
 
@@ -52,6 +90,7 @@ The goal of this project is to understand and apply advanced matrix factorizatio
    - Movie recommendations from both the **SVD** and **PMF** models.
    - Visual comparison of the SVD vs. PMF predictions for the user.
 2. Ensure real-time interaction, with recommendations and visualizations updating dynamically based on user input.
+3. The app must run successfully via: `streamlit run app.py`
 
 ### Project Repository Structure
 
@@ -72,6 +111,15 @@ matrix-factorization-project/
 │   ├── matrix_creation.py
 │   ├── recommendation.py
 │
+├── reports/
+│   ├── model_metrics.json
+│   ├── pmf_convergence.png
+│   ├── rmse_comparison.png
+│   ├── predicted_vs_actual.png
+│   ├── user_comparison.png
+│   ├── top_recommendations.png
+│   └── user_<id>_recommendations.csv
+│
 ├── app.py
 ├── requirement.txt
 ├── Movie_Recommender_System.ipynb
@@ -85,20 +133,6 @@ matrix-factorization-project/
 - **Movie_Recommender_System.ipynb**: A notebook for initial experiments, data exploration, and visualization of the model training and recommendations.
 - **README.md**: Project documentation with an overview of the recommender system, instructions for setup and running the dashboard, and additional resources.
 
-### Timeline (1-2 weeks)
-
-**Week 1:**
-
-- **Days 1-2:** Load and preprocess the dataset, create user-item interaction matrix.
-- **Days 3-4:** Implement and train the SVD model.
-- **Days 5-7:** Implement and train the PMF model, visualize MSE vs. iterations for PMF.
-
-**Week 2:**
-
-- **Days 1-2:** Compare SVD and PMF models, evaluate using MSE.
-- **Days 3-4:** Implement recommendation generation for both models.
-- **Days 5-7:** Build the Streamlit dashboard, create visualizations, and finalize the project.
-
 ### Tips
 
 Remember, a great recommender system needs to understand both the users and the content. Keep in mind the trade-off between model complexity and interpretability. Here are some additional considerations:

subjects/ai/matrix-factorization/audit/README.md

@@ -8,6 +8,14 @@
 
 ###### Is there a `requirements.txt` or `environment.yml` file listing all necessary libraries and their versions?
 
+###### Do the core files exist: `app.py`, `models/svd_model.py`, `models/pmf_model.py`, and `utils/recommendation.py`?
+
+###### Do the main dependencies import without error?
+
+```bash
+python -c "import numpy, pandas, scipy, streamlit, matplotlib"
+```
+
 ##### Data Processing and Exploratory Data Analysis
 
 ###### Is there an exploratory data analysis notebook describing insights from the MovieLens dataset?
@@ -16,6 +24,10 @@
 
 ###### Has a user-item interaction matrix been created from the data?
 
+###### Was a reproducible split used (e.g., `random_state = 42`)?
+
+###### Does the normalized user–item matrix exist at `processed/user_item_matrix.csv`
+
 ##### Matrix Factorization Models
 
 ###### Has the Singular Value Decomposition (SVD) model been implemented using scipy.sparse.linalg.svds?
@@ -24,6 +36,12 @@
 
 ###### Have both models been trained on the MovieLens dataset?
 
+###### Is the SVD predicted rating matrix saved as `reports/svd_predictions.npy`?
+
+###### Does the PMF implementation save a convergence plot (`reports/pmf_convergence.png`)?
+
+###### Are the learned factor matrices (`U`, `V`) saved (e.g., under `reports/pmf_factors/`)?
+
 ##### Model Evaluation
 
 ###### Is the Root Mean Square Error (RMSE) calculated for both models on a test set?
@@ -36,12 +54,38 @@
 
 ###### Is there a justification for when to stop training based on the learning curves?
 
+###### Does `reports/model_metrics.json` exist with fields:
+
+```json
+{
+  "SVD_RMSE": ...,
+  "PMF_RMSE": ...,
+  "PMF_vs_SVD_improvement_%": ...
+}
+```
+
+###### Are the following thresholds met?
+
+- SVD RMSE ≤ 0.90
+- PMF RMSE ≤ 0.85
+- PMF improvement ≥ 5%
+- Are the plots saved? `reports/rmse_comparison.png` and `reports/predicted_vs_actual.png`.
+
 ##### Recommendation Generation
 
 ###### Is there a function that generates movie recommendations for a user based on both SVD and PMF models?
 
 ###### Does the recommendation system return the top 10 movie recommendations for a given user?
 
+###### Does `utils/recommendation.py` expose:
+
+```python
+def generate_recommendations(user_id, model, top_n=10):
+    ...
+```
+
+###### Are user-level outputs saved as `reports/user_<id>_recommendations.csv`
+
 ##### Model Interpretability
 
 ###### Is there an analysis of the key latent factors that drive recommendations (global interpretability)?
@@ -58,12 +102,25 @@
 
 ###### For the 2 users from the training set, is there an analysis of why the recommendations were accurate for one and less accurate for the other?
 
+###### Are required visuals present in `reports/` with proper titles and labeled axes?
+
+- `pmf_convergence.png`
+- `rmse_comparison.png`
+- `predicted_vs_actual.png`
+- `user_comparison.png`
+
 ##### Streamlit Dashboard
 
 ###### Has a Streamlit dashboard been implemented?
 
 ###### Does the dashboard take a user ID as input and return recommendations and required visualizations?
 
+###### Does `streamlit run app.py` launch the dashboard successfully?
+
+###### Does the dashboard update recommendations dynamically on user ID input?
+
+###### Does it handle invalid user IDs gracefully (error shown, no crash)?
+
 ##### Additional Considerations
 
 ###### Is the code well-documented and following these good coding practices: