Skip to content

Add Graded RBP metric class#1032

Open
sushobhan2024 wants to merge 6 commits intolenskit:mainfrom
sushobhan2024:feature/graded-rbp
Open

Add Graded RBP metric class#1032
sushobhan2024 wants to merge 6 commits intolenskit:mainfrom
sushobhan2024:feature/graded-rbp

Conversation

@sushobhan2024
Copy link
Contributor

@sushobhan2024 sushobhan2024 commented Mar 5, 2026

This PR is regarding the issue #1031

  • The main change is around getting an array relevance instead of binary array.
  • RBP is now configurable with optional grade_field. The grades are assumed to be [0, 1] scaled.
  • Added a helper method graded_rank_biased_precision to compute scores using relevance and weights.
  • Added tests in existing tests for RBP

@sushobhan2024 sushobhan2024 self-assigned this Mar 5, 2026
@codecov
Copy link

codecov bot commented Mar 5, 2026

Codecov Report

❌ Patch coverage is 89.47368% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.26%. Comparing base (e5c0071) to head (11b733a).
⚠️ Report is 4 commits behind head on main.

Files with missing lines Patch % Lines
src/lenskit/metrics/ranking/_rbp.py 89.47% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1032      +/-   ##
==========================================
- Coverage   89.43%   88.26%   -1.18%     
==========================================
  Files         222      222              
  Lines       15356    15387      +31     
==========================================
- Hits        13734    13581     -153     
- Misses       1622     1806     +184     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.


grades = test.field(self.grade_field)
grade_map = dict(zip(test.ids(), grades))
relevance = np.array([grade_map.get(item, self.unknown_grade) for item in recs.ids()])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing test items get unknown_grade with a default of 0.25, and the tests also check for that same default. Was that intentional? It doesn't seem righ to me

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. It probably makes sense to split this into two tests: one that verifies the default behavior, and another that verifies the parameter is actually being used.

@samiravaez
Copy link
Contributor

Also I'm not sure if we should have a normalization option for RBP, but if it's only for experimental purposes, I think it can exceed 1 with the current default for missing test items.
To get a lower bound score I think we should consider missing items' scores as zero. If we want an upper bound we could add the residual p^k instead of assigning a fixed value.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants