Skip to content

Update benchmark results after fixing top_k/remove_seen in recommend_k_* helpers #2337

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
miguelgfierro wants to merge 16 commits into
base: staging
Choose a base branch
from
Open

Update benchmark results after fixing top_k/remove_seen in recommend_k_* helpers #2337

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
5cb939d
Remove TensorFlow from movielens benchmark notebook
miguelgfierro May 25, 2026
22d7b9a
Remove surprise/SVD from benchmark utility and notebook
miguelgfierro May 25, 2026
60e0537
Minor notebook improvements and fix embdotbias NameError
miguelgfierro May 25, 2026
9410c24
Update benchmark results and README for corrected top_k handling
miguelgfierro May 25, 2026
9e0f091
Consolidate try/except import blocks in benchmark_utils
miguelgfierro May 25, 2026
8d827d9
Fix BPR recommend_k_items: apply remove_seen before top-k selection
miguelgfierro May 25, 2026
ea9b366
Add rating threshold 3.5 to BPR deep dive and benchmark
miguelgfierro May 25, 2026
d1237f3
Use binarize() to convert ratings to implicit feedback in BPR deep dive
miguelgfierro May 25, 2026
99ddfd9
Explain unbounded prediction scores in BPR deep dive
miguelgfierro May 25, 2026
1280d4e
Use binarize() in prepare_training_bpr and rerun benchmark
miguelgfierro May 25, 2026
a0e2fb4
Fix BPR benchmark evaluation: filter test set to match training thres…
miguelgfierro May 25, 2026
f92c89e
Update BPR deep dive to max_iter=200 to match benchmark
miguelgfierro May 25, 2026
664e913
Rerun benchmark with PyTorch 2.13+cu132 (RTX 5090 sm_120 support)
miguelgfierro May 25, 2026
56bb33f
Update benchmark results in main README
miguelgfierro May 25, 2026
f1b13d0
Move RATING_THRESHOLD constant to module level at top of benchmark_utils
miguelgfierro May 25, 2026
c0aef19
Add unit tests for BPR.recommend_k_items: top_k and remove_seen
miguelgfierro May 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 8 additions & 9 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,18 +135,17 @@ Independent or incubating algorithms and utilities are candidates for the [contr

### Algorithm Comparison

We provide a [benchmark notebook](examples/06_benchmarks/movielens.ipynb) to illustrate how different algorithms could be evaluated and compared. In this notebook, the MovieLens dataset is split into training/test sets at a 75/25 ratio using a stratified split. A recommendation model is trained using each of the collaborative filtering algorithms below. We utilize empirical parameter values reported in literature [here](http://mymedialite.net/examples/datasets.html). For ranking metrics we use `k=10` (top 10 recommended items). We run the comparison on a machine with 4 CPUs, 30Gb of RAM, and 1 GPU GeForce GTX 1660 Ti with 6Gb of memory. Spark ALS is run in local standalone mode. In this table we show the results on Movielens 100k, running the algorithms for 15 epochs.
We provide a [benchmark notebook](examples/06_benchmarks/movielens.ipynb) to illustrate how different algorithms could be evaluated and compared. In this notebook, the MovieLens dataset is split into training/test sets at a 75/25 ratio using a stratified split. A recommendation model is trained using each of the collaborative filtering algorithms below. We utilize empirical parameter values reported in literature [here](http://mymedialite.net/examples/datasets.html). For ranking metrics we use `k=10` (top 10 recommended items). We run the comparison on a machine with 24 CPUs, 30Gb of RAM, and 1 GPU GeForce RTX 5090 Laptop GPU with 24Gb of memory. Spark ALS is run in local standalone mode. In this table we show the results on Movielens 100k. For BPR, ratings are binarized with a threshold of 3.5 (ratings ≥ 4 treated as positive implicit feedback) for both training and evaluation.

| Algo | MAP | nDCG@k | Precision@k | Recall@k | RMSE | MAE | R<sup>2</sup> | Explained Variance |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| [ALS](examples/00_quick_start/als_movielens.ipynb) | 0.004732 | 0.044239 | 0.048462 | 0.017796 | 0.965038 | 0.753001 | 0.255647 | 0.251648 |
| [BiVAE](examples/02_model_collaborative_filtering/cornac_bivae_deep_dive.ipynb) | 0.146126 | 0.475077 | 0.411771 | 0.219145 | N/A | N/A | N/A | N/A |
| [BPR](examples/02_model_collaborative_filtering/cornac_bpr_deep_dive.ipynb) | 0.132478 | 0.441997 | 0.388229 | 0.212522 | N/A | N/A | N/A | N/A |
| [embdotbias](examples/00_quick_start/embdotbias_movielens.ipynb) | 0.018954 | 0.117810 | 0.104242 | 0.042450 | 0.992760 | 0.776040 | 0.223344 | 0.223393 |
| [LightGCN](examples/02_model_collaborative_filtering/lightgcn_deep_dive.ipynb) | 0.088526 | 0.419846 | 0.379626 | 0.144336 | N/A | N/A | N/A | N/A |
| [NCF](examples/02_model_collaborative_filtering/ncf_deep_dive.ipynb) | 0.107720 | 0.396118 | 0.347296 | 0.180775 | N/A | N/A | N/A | N/A |
| [SAR](examples/00_quick_start/sar_movielens.ipynb) | 0.110591 | 0.382461 | 0.330753 | 0.176385 | 1.253805 | 1.048484 | -0.569363 | 0.030474 |
| [SVD](examples/02_model_collaborative_filtering/surprise_svd_deep_dive.ipynb) | 0.012873 | 0.095930 | 0.091198 | 0.032783 | 0.938681 | 0.742690 | 0.291967 | 0.291971 |
| [ALS](examples/00_quick_start/als_movielens.ipynb) | 0.010400 | 0.032800 | 0.038100 | 0.013200 | 0.966732 | 0.752552 | 0.267438 | 0.263534 |
| [BiVAE](examples/02_model_collaborative_filtering/cornac_bivae_deep_dive.ipynb) | 0.330000 | 0.471700 | 0.411300 | 0.224300 | N/A | N/A | N/A | N/A |
| [BPR](examples/02_model_collaborative_filtering/cornac_bpr_deep_dive.ipynb) | 0.229200 | 0.362300 | 0.285500 | 0.251700 | N/A | N/A | N/A | N/A |
| [embdotbias](examples/00_quick_start/embdotbias_movielens.ipynb) | 0.054900 | 0.119000 | 0.107000 | 0.042300 | 0.992760 | 0.776040 | 0.223344 | 0.223393 |
| [LightGCN](examples/02_model_collaborative_filtering/lightgcn_deep_dive.ipynb) | 0.277200 | 0.419100 | 0.359900 | 0.197100 | N/A | N/A | N/A | N/A |
| [NCF](examples/02_model_collaborative_filtering/ncf_deep_dive.ipynb) | 0.260300 | 0.393100 | 0.346900 | 0.181900 | N/A | N/A | N/A | N/A |
| [SAR](examples/00_quick_start/sar_movielens.ipynb) | 0.258500 | 0.393800 | 0.340600 | 0.185400 | N/A | N/A | N/A | N/A |

## Contributing

Expand Down
Loading