Conversation
L-M-Sherlock
changed the title
| if items.is_empty() { | ||
| return Err(FSRSError::NotEnoughData); | ||
| } | ||
| let weighted_items = recency_weighted_fsrs_items(items); |
There was a problem hiding this comment.
@L-M-Sherlock, shouldn't this be constant_weighted_fsrs_items?
In open-spaced-repetition/fsrs-optimizer#152 (comment), you said that evaluation doesn't use weights.
In addition to what @Expertium said, adding weights to the evaluation makes comparing RMSE in Anki 24.11 and 25.01 difficult.
I noticed this because Anki 24.11 shows Log loss: 0.2551, RMSE(bins): 2.91%. 64,368 reviews while Anki 25.01 shows Log loss: 0.2891, RMSE(bins): 3.18%. 64,368 reviews with the same parameters.
There was a problem hiding this comment.
It should be recency_weighted_fsrs_items otherwise Anki will prefer the old parameters.
There was a problem hiding this comment.
Well, I have changed my mind. Recency weighting makes training give more importance to recent reviews, which are those that the users care the most about.
If evaluation doesn't apply recency weighting, the more optimal parameters will produce worse metrics because the older reviews will have higher loss. So, Anki won't accept these new parameters.
But, we will need to inform users that the increased RMSE in 25.01 is due to a change in the evaluation method and this increase doesn't mean that FSRS has become worse for them.
Edit: You commented when I was typing this. 😅
There was a problem hiding this comment.
I doubt most users will check the RMSE after they update Anki. We just need to inform a small group of users if necessary.
Btw, RMSE(bins) cannot tell us much useful information. If we want to diagnose the algorithm, some calibration graphs in the section 4.2 is necessary: https://github.com/open-spaced-repetition/fsrs4anki/blob/main/fsrs4anki_optimizer.ipynb
3 participants
close #256
same to: