Online Topic Modeling with River - Representation documents

[ezerdavid]

Hi,

when doing online topic modeling with River I got the representations fine. But I noticed there are no representative documents. I also checked the code and it may be more related to the partial_fit method which don't seem to be using _save_representative_docs method.
I was wondering what's preventing the saving of representative document for partial fit?

Thanks 🙂

[MaartenGr]

That's correct! Considering the clusters change everytime you run partial_fit, the representative documents should also change. However, you have no access to any previous documents during a partial_fit from a previous run and therefore it is a bit tricky to then extract the most representative documents.

It technically needs a different procedure than _save_representative_docs as that will overwrite whatever representative documents are already there. Therefore, it would need to compare the current representative documents with the new selection and decide whether those newer ones would be better.

It is a bit out of scope for my schedule at the moment, but it does sound reasonable enough to implement. My main concern would be how to decide which metric to use to select if a new document is better than one of the previous.

[MaartenGr]

Thinking about this, it can be stupidly simple... just add the previous representative documents to the ones that you currently train on and run those through _save_representative_docs. In my head, that would only be a few lines of code.