Performance of examples

[tokafish]

I've been trying to run the large examples in the dedupe_examples project (psql_big and mysql). Both time out while building the blocking map. I was wondering if anyone could give me guidance on how long these examples are expected to take, and if I've misconfigured something on my end.

Thanks for all the work on this project!

[tokafish]

Turning on verbose mode I can see that blocking 10,000 records takes about ~100 seconds. Predicates are as follows:

INFO:dedupe.api:(TfidfTextCanopyPredicate: (0.8, name), TfidfNGramCanopyPredicate: (0.6, address), TfidfNGramCanopyPredicate: (0.6, zip))
INFO:dedupe.api:(SimplePredicate: (wholeFieldPredicate, name), SimplePredicate: (doubleMetaphone, address), SimplePredicate: (commonThreeTokens, address))
INFO:dedupe.api:(TfidfNGramCanopyPredicate: (0.8, name), SimplePredicate: (alphaNumericPredicate, zip))
INFO:dedupe.api:(SimplePredicate: (fingerprint, name), SimplePredicate: (nearIntegersPredicate, name), SimplePredicate: (fingerprint, zip))
INFO:dedupe.api:(SimplePredicate: (commonSixGram, zip), SimplePredicate: (commonIntegerPredicate, name))
INFO:dedupe.api:(SimplePredicate: (wholeFieldPredicate, address), SimplePredicate: (commonSixGram, zip), SimplePredicate: (firstTwoTokensPredicate, name))
INFO:dedupe.api:(SimplePredicate: (fingerprint, address), SimplePredicate: (commonThreeTokens, name), SimplePredicate: (commonTwoTokens, city))
INFO:dedupe.api:(TfidfNGramCanopyPredicate: (0.6, name), SimplePredicate: (commonThreeTokens, name), SimplePredicate: (commonTwoTokens, city))```

[tokafish]

Digging a bit more I find that most of the time is in invoking LevenshteinCanopyPredicate...

[fjsj]

Try disabling index predicates. It will be much faster but you'll get less recall. Set index_predicates=False in train method: https://docs.dedupe.io/en/latest/API-documentation.html#dedupe.Dedupe.train

Or try more training data.

[tokafish]

Thanks! I'll try that, but I guess I was wondering if I was doing something wrong somehow in running the example (I'm curious how long you'd expect this example to run)

[fjsj]

Last time I tested pgsql_big_dedupe_example it took only a few minutes. But you have to check how my training data compare to yours. See: dedupeio/dedupe-examples#114

[tokafish]

Thanks for the training/settings, and for all your help. This helps me get past blocking, however, I'm seeing a really slow clustering step. I am getting messages like:

WARNING:dedupe.clustering:A component contained 340599 elements. Components larger than 30000 are re-filtered. The threshold for this filtering is 7.827189083245225e-12,

which I suspect are a problem. I'm just running the psql example (from master) with your training/settings, so I'm not sure what I'm doing wrong. Any ideas?

[fgregg]

The current master branch may have some changes that will speed this up for you.