Crawling site with maxDepth > 2 causes hang

[TheTFo]

I'm crawling a small site with maxDepth === 2, and things crawl fine. As soon as up it to 3 or more, the the crawler hangs. I don't see onError or onSuccess called, or any errors. Looking through Fiddler, I do not see any request firing aside from the first batch. How should I troubleshoot this?

What is the current behavior?
Crawler seems to hang with no error when maxDepth > 2. It's a rather small site.

If the current behavior is a bug, please provide the steps to reproduce
Setting up to queue a particular site with maxDepth > 2 causes things to hang

What is the expected behavior?
With any depth, the result should be the same.

Please tell us about your environment:

• Version: 1.8.0
• Platform / OS version: MacOS 11
• Node.js version: 12.8.4

[michaelpapesch]

i had the same problem, it was CPU usage for me on a server with 2 cores.
with 8 cores i had no problems. maybe that helps.

[iamprageeth]

You can try adding more memory to node --max_old_space_size=4000

[goldingdamien]

I had a similar issue, and fixed it with "maxConcurrency": 1.