#11 Crawling facebook.com causes infinite crawling after /careers/help page

[1-ashraful-islam]

When crawling facebook.com the crawler goes into infinite crawling where /help keeps getting appended to the url. When manually browsing the link, it redirects to metacareers.com but not while crawling with fetch command.

Crawling https://facebook.com/careers/?ref=pf
Crawling https://facebook.com/careers/help/637205020878504
Crawling https://facebook.com/careers/help/help/637205020878504
Crawling https://facebook.com/careers/help/help/help/637205020878504
Crawling https://facebook.com/careers/help/help/help/help/637205020878504

[1-ashraful-islam]

This issue also happens with other facebook links. If you start crawling from https://facebook.com/careers it runs into this:

Crawling https://facebook.com/biz/directory/help/637205020878504
Crawling https://facebook.com/biz/directory/help/help/637205020878504
Crawling https://facebook.com/biz/directory/help/help/help/637205020878504