Quantifying memorization progress

[SplittyDev]

Hey, what would be the best way to quantify the progress in order to tell the user something like "Your mastery of this card is x%"?

At first I thought the retrievability $R$ would be a good candidate because $0 \leq R \leq 1$. But even though it's a good value to turn into a percentage due to its normalized nature, it doesn't accurately reflect the "mastery" of the card, since the recall will always be high shortly after a review session.

I then thought that $S$ could be a good value to use, but it's not bounded or normalized in any sensible way that would make it suitable for quantifying the learning progress.

For lack of a better option, I was calculating the "mastery" value as $M = \frac{\max(S, 365)}{365}$ and assuming that $S \geq 365 \implies R > 0.9$ is good enough to qualify as having mastered the card, even though it's not technically correct.

I've now switched to calculating mastery as retrievability(for date: now + 1 year) (pseudocode), since the numbers seem more sensible to me than the previous stability method, but it still suffers from some of the same issues.

Since it's quite important to show the user a sensible approximation of their progress for the sake of motivation, it would be great to know how to best approach this.

[Expertium]

I'm not sure if showing the user a measure of "mastery" is a good idea to begin with. FSRS is already quite complex, stability and retrievability are confusing for a lot of people. I would say that stability itself is a good measure of mastery. Also, it's technically bounded: in FSRS, the upper limit is 36500 days aka 100 years; though I'd argue that 10 years is already mastery. I don't doubt that it's possible to come up with a clever metric that uses clever math, I doubt that most users' reaction would be anything other than "Oh, it's another mysterious FSRS number, whatever".

Btw, while this isn't what you're looking for, there is a metric that can be compared across different people: knowledge acquisition rate. Install the Helper add-on and Shift + Left Mouse Click on Stats to find it.

[SplittyDev]

Thanks for the answer!

I'm developing a language self-learning app, which uses FSRS as the backing algorithm.

My app has a "word bank" with a list of all words, and these words have attached FSRS cards, but that's just an implementation detail the user neither knows nor cares about. FSRS and the card system are used for flashcard review, but the users don't concern themselves with the details of that. They just go to the review tab and go through the cards.

Since my app is targeting casual learners and not exactly the same crowd that uses Anki with custom plugins, settings, algorithms and parameters, I want the users to have some kind of idea how their learning is going.

Not showing anything isn't gonna make them feel like they're making progress, and gamifying this experience a bit is surely gonna help people stay motivated in the long run. Seeing continual progress is always a good motivator.

I've switched to a tiered metric now:

• First, use the 1-year retrievability to show progress.
• Once that reaches 90%+, give the user an "upgrade" to the next tier and use the 2-year retrievability
• Same thing again, once it reaches 90%, user gets an "upgrade" to 5-year retrievability
• And finally, once that is done, the final mastery tier is reached, which uses the 10-year retrievability

The progress is nicely and colorfully rendered next to each word, so the user can see "this word has a nice and green badge with two stars, and I'm close to reaching the third", gamifying the experience and keeping them motivated to learn more.

The flashcard system in the app will keep working regardless of the calculated mastery. The user will always be able to learn more and improve their retrievability way beyond the app's visual 10-year cap, but I strongly believe that the system is gonna help people keep the learning up and not give up because they don't feel any progress.

You could compare this system to Memrise, which uses the growing stages of plants to signify progress, or Duolingo, which uses a star-system. Each of those systems might not perfectly quantify the user's knowledge/memory journey, but they're a big factor in why these apps are popular.

I'm just trying to find a system that works well with FSRS and gives the user a decent approximation of their progress, ideally without gaslighting them into thinking they've "mastered" a word when they really really haven't.

I'd argue that having 90%+ successful recall probability in 10 years from now is quite the good indicator of learning progress, but using that as the initial number would discourage users, since they barely see any progress. The tiered system is the best I could come up with so far, but I'd appreciate different ideas a lot.

In the end, I'm just trying to make learning fun, in addition to being efficient and actually helping the user on their language learning journey without actively and purposefully misleading them (as some other apps are certainly doing).

[Expertium]

I see. Yeah, that's a different situation. I like your idea with tiered retrievability, maybe I'll try to think of something too.

Btw, in your app will parameters be optimized automatically and periodically?

[SplittyDev]

I'm thinking of doing that, but I'll have to look a bit deeper into how much that actually helps and how hard it'll be to implement.

I've already done the work of porting the Swift algorithm to FSRS 4.5 from 4.0 (and upstreamed it), but porting the optimizer too might be a little bit too much for me at the moment 😄

From what I have gathered so far, I'd assume that the default parameters are good enough^TM for the average user, since they're derived from a fairly large data set. I'm also not sure how much additional efficiency there is to be gained from doing automatic optimization, or if it could even hurt and result in regressed performance in the long term.

[ishiko732]

Currently, the fsrs-browser can be used for parameter optimization in webview that support wasm and worker, but I'm not sure if Swift can be embedded in the webview to execute. In my tests, both Safari and Chrome were able to run successfully. Currently, it is still a bit difficult to simplify the calling of parameter optimization on the web. If you want to understand the implementation details, you can check the related code in the ts-fsrs-demo. https://github.com/ishiko732/ts-fsrs-demo/tree/main/src/components/train

[Expertium]

or if it could even hurt and result in regressed performance in the long term.

No, that's very unlikely. The more reviews the user does, the more data there is for the optimizer to work with, the better it can adapt to this user. It's the lack of optimization that would hurt, especially in cases where there is plenty of review data (like, tens of thousand of reviews) and all that data is unused. According to the benchmark, optimized FSRS works better than unoptimized FSRS pretty much universally.
Also, FSRS-5 is out.

[joshdavham]

Late to this conversation, but I think that trying to find a reasonable-ish way to quantify memory progress in a way that learners feel that they understand is very important. I specifically work in the field of "immersion learning" and one of the biggest pain points of immersion learners is feeling like they're not making progress, so finding a way to show them that they actually are making progress is super duper important.

It was actually a funny realization I had about a year ago, but it was that one of the primary functions of anki for an immersion learner is gamification. They get to see their numbers go up, they get to graduate cards and they can even get some pretty graphs. Without that, they feel like they're not making progress.

[L-M-Sherlock]

Here is a repo you may be interested in:

https://github.com/xiety/AnkiHistoryVisualization

[koljapluemer]

Even later to this conversation, but I had to think about this recently while using fsrs for a geography learning game (this, for context) so here is my two cents:

1. I think you can use any number of fairly simple metrics to represent the memorization process; for my use case simply counting how many times in a row the user got an item right was sufficient, although I was tempted to implement the tiered retrievability approach detailed above.
2. What I think is more important: The way people generally think about mastering/learning something is that mastery is not lost again after being achieved. If a learning app tells me today that I memorized flashcard XYZ on "10 year level" or "gold level" or whatever, and then tomorrow I get the flashcard wrong (and thus retrievability goes down, or in my case the streak is broken), and then I lose that mastery level again, that feels really weird. Life usually doesn't work this way: Once you're a grandmaster in chess you keep that title even if you play worse chess next year. So if we're talking motivation/gamification, I think it's best that once a word is "mastered", it stays mastered. If you then forget the word, it may be the best idea to communicate something like "ok yeah you still memorized/mastered this word but it's time for you to refresh that title" or something. I think duolingo level mastery works like that — worth stealing in this context IMO.

anyways hope that's helpful for someone :)

[Expertium]

@koljapluemer @ishiko732 if you are using FSRS, a measure of mastery based on stability rather than retrievability makes a lot more sense. It's great if the user can recall the card with a 99% chance today, but if that drops to 50% tomorrow, suddenly, it's not so great. Stability - defined as "the number of days it takes for retrievability to fall from 10% to 90%" in FSRS - also grows with the number of successful reps. Loosely speaking, stability is "how long does it take you to forget this material". It's kinda like half-life in physics.

The "problem" (from koljapluemer's perspective) is that stability drops really hard after a failed review aka a lapse. It can drop from 1000 days to 10 days, for example. So there is a contradiction between what is considered intuitive and how memory actually behaves (assuming FSRS isn't ridiculously, wildly inaccurate). If you want a measure of mastery that doesn't plummet when the user presses Again/Fail/Forgot/[whatever is the name of the fail button in your app] and monotonically grows over time/over the number of reviews, well...it doesn't exist. At least not if you are using FSRS.

[L-M-Sherlock]

What about the average (due date - current date)? It increases exponentially and decays linearly. @Luc-Mcgrady would you like to test this idea?

[koljapluemer]

@Expertium Good points, thanks, I hadn't really reflected on using stability vs. retrievability.

As for the measure not existing, my point was that as developer of a learning app it may be in your interest to ignore whether a given value falls again after you reach a threshold, and simply let it stay on "yep, mastered" no matter what. Just like in Judo, where you don't loose your black belt even if you break both your feet and couldn't win a fight against anyone :)

Goal here is not accurate prediction of cognitive variables, but "human-readable" feedback on how the user's learning journey goes

[koljapluemer]

@L-M-Sherlock wouldn't we hit a weird state when the due date is in the past?

Also I think that measure would lead to you loosing all your "mastered" "titles" when you open the learning app in question after taking a 3 months break, no? But maybe I'm misunderstanding what you're proposing or the use case 🤔

[L-M-Sherlock]

To be honest, I'd forgotten most of math within months after I graduated.