Website request limits

[arlieu]

StockCrawler receives delayed or fast-failed responses after a website's request thresholds is reached by the IP requests is using.

Requirements:

• Detect delayed responses
• Change IP address
• Send requests from new IP address
• Spoof headers

[arlieu]

Need to add spoofing for intelligent crawl detectors

[arlieu]

Header spoofing added in feat-header-spoofing branch. Web server picks up on forged header currently. Need to research HTTP headers and norms.