Love what you have built here! I've been reading through the code trying to get a better sense for how everything works. I noticed that in rate.py you deduplicate text by indexing with a 32 bit hash. This works well for small samples since the risk of a collision for two texts is around 1 in 4.3 billion. However, for larger sample sizes the odds of at least one collision go up rapidly. You get to a 50% chance of at least one collision with a sample size of around 65 thousand, and if you were analyzing 10,000,000 tweets your expected number of collisions would be over 11,000. Using a 64 bit has would take the expected number of collisions to less than 0.000003.
If you think that is worth changing I am more than happy to submit a pull request.
Love what you have built here! I've been reading through the code trying to get a better sense for how everything works. I noticed that in rate.py you deduplicate text by indexing with a 32 bit hash. This works well for small samples since the risk of a collision for two texts is around 1 in 4.3 billion. However, for larger sample sizes the odds of at least one collision go up rapidly. You get to a 50% chance of at least one collision with a sample size of around 65 thousand, and if you were analyzing 10,000,000 tweets your expected number of collisions would be over 11,000. Using a 64 bit has would take the expected number of collisions to less than 0.000003.
If you think that is worth changing I am more than happy to submit a pull request.