Feature/cleanup fuzzyreplace

[jaderabbit]

• Replaced fuzzywuzzy with the C++ based rapidfuzz
• Added a progress bar for fuzzywuzzy
• Moved all the installations to the top cell

[review-notebook-app]

Check out this pull request on 

Review Jupyter notebook visual diffs & provide feedback on notebooks.

---

Powered by ReviewNB

[espoirMur]

This looks awesome @JADE,

did we get the time. to benchmark it?

How long does it take on a medium-size dataset?

[espoirMur]

@jaderabbit and @juliakreutzer is this the right way to install joeynmt?

like pip install joeynmt

I remember last time I tried it it gave me headaches and I was forced to install it from Github...

using pip install git+https://github.com/joeynmt/joeynmt.git

also it is missing the version...

[jaderabbit]

@espoirMur I added it to pypi a few weeks ago so this should be fine now - I still need to update the folder locations later down so we don't need to install joeynmt via cloning it.

Yes, I should do versions for all the requirements - thanks for the push!

[espoirMur]

if it is on PyPI that is cool....

[jaderabbit]

This looks awesome @JADE,

did we get the time. to benchmark it?

How long does it take on a medium-size dataset?

@espoirMur I'm actually going to change it.

• The benchmark I did do for fuzzywuzzy vs rapidfuzz though using timeit: Where fuzzywuzzy takes 1s for 100 patterns, rapidfuzz takes 32ms 🎉
• On the other hand: The new progress bar I added slows EXTENSIVELY - in fact, so far, your approach (which technically has no logging at all) is the fastest still. I might rewrite using a slightly different multiprocessing library to allow for some sense of output - or at a progress bar which updates based on an estimation or something.

[espoirMur]

LGTM

[espoirMur]

This looks awesome @JADE,
did we get the time. to benchmark it?
How long does it take on a medium-size dataset?

@espoirMur I'm actually going to change it.

• The benchmark I did do for fuzzywuzzy vs rapidfuzz though using timeit: Where fuzzywuzzy takes 1s for 100 patterns, rapidfuzz takes 32ms 🎉
• On the other hand: The new progress bar I added slows EXTENSIVELY - in fact, so far, your approach (which technically has no logging at all) is the fastest still. I might rewrite using a slightly different multiprocessing library to allow for some sense of output - or at a progress bar which updates based on an estimation or something.

If that is the speed we got .... it's really really nice....

[juliakreutzer]

That's an awesome improvement, thank you @jaderabbit! 💯
Regarding the progress bar: do you want to keep it for now and merge or wait for the merge until there's a better solution?

[juliakreutzer]

@jaderabbit shall we merge this? Or not because of the progress bar?