Is the pipeline blocking? #41
Description
It looks like the scrapy-mongodb pipeline is blocking. So the crawl speed is limited to the mongodb data write speed.
Is it better to change it to non-blocking pipeline such as using txmongo?
{{ message }}
It looks like the scrapy-mongodb pipeline is blocking. So the crawl speed is limited to the mongodb data write speed.
Is it better to change it to non-blocking pipeline such as using txmongo?