Check robots.txt for crawl delays #16

Open

Open

Check robots.txt for crawl delays#16

Assignees

Labels

enhancementimportant

Milestone

Use robotstxt::get_robotstxt
Example

See also #14

Metadata

Assignees

briatte

Labels

enhancementimportant

Projects

No projects

Milestone

v.3.0

Relationships

None yet

Development

No branches or pull requests