Skip to content

docs: label MAP@k notebook outputs consistently #2350

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
he-yufeng wants to merge 1 commit into
base: staging
Choose a base branch
from
Open

docs: label MAP@k notebook outputs consistently #2350

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions examples/00_quick_start/als_movielens.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -416,7 +416,7 @@
"text": [
"Model:\tALS\n",
"Top K:\t10\n",
"MAP:\t0.006527\n",
"MAP@K:\t0.006527\n",
"NDCG:\t0.051718\n",
"Precision@K:\t0.051274\n",
"Recall@K:\t0.018840\n"
Expand All @@ -434,7 +434,7 @@
"source": [
"print(\"Model:\\tALS\",\n",
" \"Top K:\\t%d\" % rank_eval.k,\n",
" \"MAP:\\t%f\" % rank_eval.map_at_k(),\n",
" \"MAP@K:\\t%f\" % rank_eval.map_at_k(),\n",
" \"NDCG:\\t%f\" % rank_eval.ndcg_at_k(),\n",
" \"Precision@K:\\t%f\" % rank_eval.precision_at_k(),\n",
" \"Recall@K:\\t%f\" % rank_eval.recall_at_k(), sep='\\n')"
Expand Down
4 changes: 2 additions & 2 deletions examples/02_model_collaborative_filtering/als_deep_dive.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -502,7 +502,7 @@
"Precision@k = 0.04379639448568401\n",
"Recall@k = 0.014286194686756822\n",
"NDCG@k = 0.03730295615527768\n",
"Mean average precision = 0.0034619726118607337\n"
"MAP@k = 0.0034619726118607337\n"
]
},
{
Expand All @@ -529,7 +529,7 @@
" \"Precision@k = {}\".format(evaluations.precision_at_k()),\n",
" \"Recall@k = {}\".format(evaluations.recall_at_k()),\n",
" \"NDCG@k = {}\".format(evaluations.ndcg_at_k()),\n",
" \"Mean average precision = {}\".format(evaluations.map_at_k()),\n",
" \"MAP@k = {}\".format(evaluations.map_at_k()),\n",
" sep=\"\\n\"\n",
")"
]
Expand Down
6 changes: 3 additions & 3 deletions examples/02_model_collaborative_filtering/surprise_svd_deep_dive.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -587,7 +587,7 @@
"\n",
"As we can see, the RMSE and MAE (Mean Absolute Error) are pretty low (i.e. good), indicating that on average the error in the predicted ratings is less than 1. The RMSE is of course a bit higher, because high errors are penalized much more.\n",
"\n",
"For comparison with other models, we also display Top-k and ranking metrics (MAP, NDCG, etc.). Note however that the SVD algorithm was designed for achieving high accuracy, not for top-rank predictions."
"For comparison with other models, we also display Top-k and ranking metrics (MAP@K, NDCG, etc.). Note however that the SVD algorithm was designed for achieving high accuracy, not for top-rank predictions."
]
},
{
Expand All @@ -604,7 +604,7 @@
"rsquared:\t0.288045\n",
"exp var:\t0.288157\n",
"----\n",
"MAP:\t\t0.051501\n",
"MAP@K:\t\t0.051501\n",
"NDCG:\t\t0.110465\n",
"Precision@K:\t0.100425\n",
"Recall@K:\t0.035267\n"
Expand Down Expand Up @@ -636,7 +636,7 @@
"print(\"----\")\n",
"\n",
"print(\n",
" \"MAP:\\t\\t%f\" % eval_map,\n",
" \"MAP@K:\\t\\t%f\" % eval_map,\n",
" \"NDCG:\\t\\t%f\" % eval_ndcg,\n",
" \"Precision@K:\\t%f\" % eval_precision,\n",
" \"Recall@K:\\t%f\" % eval_recall,\n",
Expand Down
12 changes: 6 additions & 6 deletions examples/03_evaluate/als_movielens_diversity_metrics.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@
" - We also define a process which randomly recommends unseen items to each user. \n",
" - We show two options to calculate item-item similarity: (1) based on item co-occurrence count; and (2) based on item feature vectors.\n",
" \n",
"The comparision results show that the ALS recommender outperforms the random recommender on ranking metrics (Precision@k, Recall@k, NDCG@k, and\tMean average precision), while the random recommender outperforms ALS recommender on diversity metrics. This is because ALS is optimized for estimating the item rating as accurate as possible, therefore it performs well on accuracy metrics including rating and ranking metrics. As a side effect, the items being recommended tend to be popular items, which are the items mostly sold or viewed. It leaves the [long-tail items](https://github.com/microsoft/recommenders/blob/main/GLOSSARY.md) having less chance to get introduced to the users. This is the reason why ALS is not performing as well as a random recommender on diversity metrics. \n",
"The comparision results show that the ALS recommender outperforms the random recommender on ranking metrics (Precision@k, Recall@k, NDCG@k, and\tMAP@k), while the random recommender outperforms ALS recommender on diversity metrics. This is because ALS is optimized for estimating the item rating as accurate as possible, therefore it performs well on accuracy metrics including rating and ranking metrics. As a side effect, the items being recommended tend to be popular items, which are the items mostly sold or viewed. It leaves the [long-tail items](https://github.com/microsoft/recommenders/blob/main/GLOSSARY.md) having less chance to get introduced to the users. This is the reason why ALS is not performing as well as a random recommender on diversity metrics. \n",
"\n",
"From the algorithmic point of view, items in the tail suffer from the cold-start problem, making them hard for recommendation systems to use. However, from the business point of view, oftentimes the items in the tail can be highly profitable, since, depending on supply, business can apply a higher margin to them. Recommendation systems that optimize metrics like novelty and diversity, can help to find users willing to get these long tail items. Usually there is a trade-off between one type of metric vs. another. One should decide which set of metrics to optimize based on business scenarios."
]
Expand Down Expand Up @@ -576,7 +576,7 @@
" \"Precision@k\": ranking_eval.precision_at_k(),\n",
" \"Recall@k\": ranking_eval.recall_at_k(),\n",
" \"NDCG@k\": ranking_eval.ndcg_at_k(),\n",
" \"Mean average precision\": ranking_eval.map_at_k()\n",
" \"MAP@k\": ranking_eval.map_at_k()\n",
" \n",
" }\n",
" return metrics \n",
Expand Down Expand Up @@ -761,7 +761,7 @@
"metadata": {},
"outputs": [],
"source": [
"cols = [\"Data\", \"Algo\", \"K\", \"Precision@k\", \"Recall@k\", \"NDCG@k\", \"Mean average precision\",\"catalog_coverage\", \"distributional_coverage\",\"novelty\", \"diversity\", \"serendipity\" ]\n",
"cols = [\"Data\", \"Algo\", \"K\", \"Precision@k\", \"Recall@k\", \"NDCG@k\", \"MAP@k\",\"catalog_coverage\", \"distributional_coverage\",\"novelty\", \"diversity\", \"serendipity\" ]\n",
"df_results = pd.DataFrame(columns=cols)\n",
"\n",
"df_results.loc[1] = als_results \n",
Expand Down Expand Up @@ -800,7 +800,7 @@
" <th>Precision@k</th>\n",
" <th>Recall@k</th>\n",
" <th>NDCG@k</th>\n",
" <th>Mean average precision</th>\n",
" <th>MAP@k</th>\n",
" <th>catalog_coverage</th>\n",
" <th>distributional_coverage</th>\n",
" <th>novelty</th>\n",
Expand Down Expand Up @@ -844,7 +844,7 @@
"</div>"
],
"text/plain": [
" Data Algo K Precision@k Recall@k NDCG@k Mean average precision \\\n",
" Data Algo K Precision@k Recall@k NDCG@k MAP@k \\\n",
"1 100k als 10 0.044374 0.015567 0.040657 0.004202 \n",
"2 100k random 10 0.018259 0.006516 0.018537 0.002038 \n",
"\n",
Expand All @@ -871,7 +871,7 @@
"metadata": {},
"source": [
"#### Conclusion\n",
"The comparision results show that the ALS recommender outperforms the random recommender on ranking metrics (Precision@k, Recall@k, NDCG@k, and\tMean average precision), while the random recommender outperforms ALS recommender on diversity metrics. This is because ALS is optimized for estimating the item rating as accurate as possible, therefore it performs well on accuracy metrics including rating and ranking metrics. As a side effect, the items being recommended tend to be popular items, which are the items mostly sold or viewed. It leaves the long-tail less popular items having less chance to get introduced to the users. This is the reason why ALS is not performing as well as a random recommender on diversity metrics. "
"The comparision results show that the ALS recommender outperforms the random recommender on ranking metrics (Precision@k, Recall@k, NDCG@k, and\tMAP@k), while the random recommender outperforms ALS recommender on diversity metrics. This is because ALS is optimized for estimating the item rating as accurate as possible, therefore it performs well on accuracy metrics including rating and ranking metrics. As a side effect, the items being recommended tend to be popular items, which are the items mostly sold or viewed. It leaves the long-tail less popular items having less chance to get introduced to the users. This is the reason why ALS is not performing as well as a random recommender on diversity metrics. "
]
},
{
Expand Down
8 changes: 4 additions & 4 deletions examples/05_operationalize/als_movie_o16n.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -633,7 +633,7 @@
"source": [
"### 2.5 Evaluate how well ALS performs\n",
"\n",
"Evaluate model performance using metrics such as Precision@K, Recall@K, [MAP@K](https://en.wikipedia.org/wiki/Evaluation_measures_\\(information_retrieval\\) or [nDCG@K](https://en.wikipedia.org/wiki/Discounted_cumulative_gain). For a full guide on what metrics to evaluate your recommender with, consult [this guide]../03_evaluate/evaluation.ipynb)."
"Evaluate model performance using metrics such as Precision@K, Recall@K, [MAP@K](https://en.wikipedia.org/wiki/Evaluation_measures_\\(information_retrieval\\)) or [nDCG@K](https://en.wikipedia.org/wiki/Discounted_cumulative_gain). For a full guide on what metrics to evaluate your recommender with, consult [this guide](../03_evaluate/evaluation.ipynb)."
],
"metadata": {}
},
Expand Down Expand Up @@ -701,7 +701,7 @@
"print(\n",
" \"Model:\\tALS\",\n",
" \"Top K:\\t%d\" % rank_eval.k,\n",
" \"MAP:\\t%f\" % rank_eval.map_at_k(),\n",
" \"MAP@K:\\t%f\" % rank_eval.map_at_k(),\n",
" \"NDCG:\\t%f\" % rank_eval.ndcg_at_k(),\n",
" \"Precision@K:\\t%f\" % rank_eval.precision_at_k(),\n",
" \"Recall@K:\\t%f\" % rank_eval.recall_at_k(), sep='\\n'\n",
Expand All @@ -714,7 +714,7 @@
"text": [
"Model:\tALS\n",
"Top K:\t10\n",
"MAP:\t0.003698\n",
"MAP@K:\t0.003698\n",
"NDCG:\t0.034331\n",
"Precision@K:\t0.039343\n",
"Recall@K:\t0.014976\n"
Expand Down Expand Up @@ -1235,4 +1235,4 @@
},
"nbformat": 4,
"nbformat_minor": 1
}
}
Loading