ai-crawlers

Answer Engine Optimization for the modern web. Make your site discoverable by ChatGPT, Claude, Perplexity & AI search engines. Generates llms.txt, robots.txt, sitemap, JSON-LD & more.

Automate web workflows on real browser tabs without hosting a browser farm.

Curated GEO platforms, repositories, and verification scripts for AI-era discoverability.

Automatically generate llms.txt and llms-full.txt files for Laravel applications to help AI models and LLM crawlers understand your website.

Scope, audit, harden, and retest web projects with AI coding agents and reproducible evidence.

🤖 Generate optimized robots.txt files for AI search engine crawlers (GPTBot, PerplexityBot, ClaudeBot, and more)

See your site the way search engines and AI see it.

Server-side AI crawler & bot traffic tracking for Laravel — reports ChatGPT, Claude, Googlebot, GPTBot & co. to DataFast. Unofficial PHP port of @datafast/ai-crawl.

Claude Code skill that generates and audits ai.txt, robots.txt (AI-crawler rules), llms.txt, and TDM opt-out files for websites.

AI-Crawler-Blocklist publishes AI crawler blocklists and deployment-ready firewall snippets from official operator-published sources. It separates verified IP ranges, user-agent rules, robots.txt controls, and watch lists so site operators can choose the right enforcement level without mixing signal quality.

Complete database of AI search engine crawler user-agents (GPTBot, PerplexityBot, ClaudeBot) with robots.txt configuration examples

LogWerk is a lightweight, offline-first server log analyzer dashboard that runs entirely in your web browser.

Smart prerendering for WordPress — make your content visible to search engines, social bots, and AI crawlers. Zero config. 29+ bots. ~200ms.

GEO SEO agent skill for AI search optimization, answer engine optimization, AI search visibility, technical SEO, structured data, citations, and AI crawler access.

Define AI permissions once. Emit the rules today's AI web actually uses. One manifest → robots.txt, AIPREF headers, X-Robots-Tag, Google-Extended, TDMRep.

Free, deterministic GEO audit that scores whether AI answer engines (ChatGPT, Perplexity, Claude, Google AI Overviews) can retrieve, read, and quote your website

MOVED: This tool is now part of the aah.monster ecosystem.

Curated list of AI visibility / GEO / AEO resources: tools, AI crawler datasets, agent skills, llms.txt, MCP servers, research papers, and guides.

MCP server for Googlebot vs AI crawler parity analysis — compare crawl behaviour from Nginx logs with GSC data to detect visibility gaps

Edge-native Akamai EdgeWorker that intercepts AI crawler traffic and serves token-efficient Markdown instead of HTML — no origin changes required