Add robot.txt support

[dnielsen]

Is there an existing issue for this?

• I have searched the existing issues

Task/Feature/Issue Details

We want to observe the expectations of every website we scrape. So let's check for robot.txt

Additional Context

No response