Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1032 +/- ##
==========================================
- Coverage 89.43% 88.26% -1.18%
==========================================
Files 222 222
Lines 15356 15387 +31
==========================================
- Hits 13734 13581 -153
- Misses 1622 1806 +184 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
|
||
| grades = test.field(self.grade_field) | ||
| grade_map = dict(zip(test.ids(), grades)) | ||
| relevance = np.array([grade_map.get(item, self.unknown_grade) for item in recs.ids()]) |
There was a problem hiding this comment.
Missing test items get unknown_grade with a default of 0.25, and the tests also check for that same default. Was that intentional? It doesn't seem righ to me
There was a problem hiding this comment.
Good point. It probably makes sense to split this into two tests: one that verifies the default behavior, and another that verifies the parameter is actually being used.
|
Also I'm not sure if we should have a normalization option for RBP, but if it's only for experimental purposes, I think it can exceed 1 with the current default for missing test items. |
This PR is regarding the issue #1031
relevanceinstead of binary array.RBPis now configurable with optionalgrade_field. The grades are assumed to be [0, 1] scaled.graded_rank_biased_precisionto compute scores using relevance and weights.RBP