robots-txt

Polite, slim and concurrent web crawler.

advertools - online marketing productivity and analysis tools

🤖 The largest directory for AI-ready documentation and tools implementing the proposed llms.txt standard

A simple and flexible web crawler that follows the robots.txt policies and crawl delays.

Tame the robots crawling and indexing your Nuxt site.

🐘 Simple PHP library to help developers 🍻 do better on-page SEO optimization 🤖

The robots.txt exclusion protocol implementation for Go language

A set of reusable Java components that implement functionality common to any web crawler

Determine if a page may be crawled from robots.txt, robots meta tags and robot headers

Ultimate Website Sitemap Parser

A simple but powerful web crawler library for .NET

Open-Source Python Based SEO Web Crawler

Opt-Out tool to check Copyright reservations in a way that even machines can understand.

NodeJS robots.txt parser with support for wildcard (*) matching.

Known tags and settings suggested to opt out of having your content used for AI training.

Parse through any sitemap in Node.js

Makes it easy to add robots.txt, sitemap and web app manifest during build to your Astro app.

Open-source GEO / AEO skill for AI visibility readiness: check robots.txt, llms.txt, schema, citation signals, competitor gaps, and buyer prompts

Answer Engine Optimization for the modern web. Make your site discoverable by ChatGPT, Claude, Perplexity & AI search engines. Generates llms.txt, robots.txt, sitemap, JSON-LD & more.

grobotstxt is a native Go port of Google's robots.txt parser and matcher library.