Robots.txt Checker
One file decides who may read your site — and one wrong line blocks a crawler you wanted. Enter a domain: we fetch the live robots.txt and give a verdict per bot, search engines and AI crawlers side by side, plus the syntax problems crawlers silently ignore.
How verdicts are decided
The parser applies the same rules crawlers do: a bot named in its own group follows that group and ignores the * group; a bot not named falls back to *; and a group disallowing / blocks the whole site unless an Allow rule opens it back up. Consecutive User-agent lines share one group, exactly per spec.
The mistakes we flag
Rules placed before any User-agent line (ignored), a bare Host: directive (read by nobody), unknown directives (typos crawl right past), and a missing Sitemap line — the one robots.txt job that helps you even when you block nothing.
More free tools
All toolsAccess is step one. Being cited is the game.
The full SeoVision audit checks your whole crawl path and then measures what Google and every AI assistant actually say about your brand.
Get your free audit