Skip to content

Use FST instead of Trie for consuming less memory #10

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Conversation

mominul
Copy link
Member

@mominul mominul commented Jun 10, 2025

This makes upodesh to use FST instead of Trie which significantly reduces the amount of memory consumption (from 140MB to 2MB), a 98.57% reduction! 🎉

But the runtime performance gets a hit, but I think we can bear this tradeoff in the long run.

Test Case upodesh (mean µs) upodesh-fst (mean µs)
a ~2.134 ~3.341
arO ~6.971 ~11.840
bistari ~4.628 ~9.734

But the fst version of upodesh is still significantly faster than the regex-based implementation!

Test Case upodesh-fst (µs) regex (µs) Difference
a ~3.341 ~194.34 🚀 fst is ~58× faster
arO ~11.840 ~246.53 🚀 fst is ~20.8× faster
bistari ~9.734 ~353.74 🚀 fst is ~36.3× faster

cc @gulshan @mugli

Closes #3

@mugli
Copy link

mugli commented Jun 10, 2025

OMG! 😍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Huge amount of memory allocation
2 participants