Generate custom robots.txt files from your sitemap.xml.
BotBlock takes in your sitemap and allows you to generate a tailored robots.txt file that only blocks certain pages, file types, and crawlers.
sidnarsipur/BotBlock
Folders and files
| Name | Name | Last commit date | ||
|---|---|---|---|---|