AI Crawler Checker
If AI crawlers can't read your site, AI assistants can't cite it. Enter a domain to see the live robots.txt verdict for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended — plus whether your sitemap and llms.txt are in place.
What this checks
The tool fetches your live robots.txt and parses it per user-agent, exactly the way each crawler does: a crawler named with Disallow: / is blocked, a crawler not named falls back to the site-wide rule, and everything else is allowed. It also counts the URLs in your sitemap and checks for an llms.txt.
Why it matters
Being crawlable is step one of AI visibility: a blocked GPTBot means ChatGPT's search never reads your pages, and a missing sitemap means crawlers discover your content late or not at all. Whether assistants actually mention and cite your brand — and what they say — is what our full audit measures.
More free tools
All toolsCrawlable is not the same as cited
Run the free audit to see how ChatGPT, Claude, Gemini and Perplexity actually answer questions about your market — and whether you show up.
Get your free audit