Can AI crawlers read your site?
Enter your website and see which AI crawlers your robots.txt lets in, from ChatGPT and Claude to Perplexity and Gemini, with the lines to change if you're blocking one by mistake.
We read your robots.txt and check 13 AI crawlers. It takes a few seconds.
What this checks
Every AI assistant sends its own crawlers. Some collect pages to train models, some fetch pages to answer a question with search, and some fetch a page when a user pastes a link. Each one reads your robots.txt first and follows the rules written for it, or the rules for every crawler (User-agent: *) when there are none.
The checker reads your robots.txt the way those crawlers do and tells you, crawler by crawler, whether it may read your home page.
Why it matters
An assistant that can't read your site can't quote it or link to it. Many sites block AI crawlers without meaning to: a security plugin, a template robots.txt or a host's default setting adds a blanket rule, and the site quietly drops out of AI answers.
Blocking training crawlers while allowing search crawlers is a reasonable choice. Blocking the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) means you won't be cited when people ask those assistants for recommendations.
Questions, answered
How do I allow GPTBot and ClaudeBot in robots.txt?
Remove any Disallow: / line in a group that names them, or in the User-agent: * group if they have no group of their own. To be explicit, add a group with User-agent: GPTBot (or ClaudeBot) followed by Allow: /.
Should I block AI crawlers?
It depends on whether you want to be recommended. If AI answers are a source of customers, let the search and user crawlers in. You can still block training-only crawlers if you don't want your content used to train models.
Does blocking Google-Extended affect my Google rankings?
No. Google-Extended only controls whether Google may use your content for Gemini and its AI models. Google Search crawls with Googlebot, which is a separate rule.
How quickly do changes to robots.txt take effect?
Crawlers usually re-read robots.txt within a day. Getting pages crawled and showing up in AI answers takes longer, from days to a few weeks.
Keep going
AI recommendation checker
Ask Claude a customer's question with web search and see whether it names you, and who it names instead.
Schema markup generator
Fill in a form and copy JSON-LD for your organization, local business, product or FAQs.
Rather have it all done for you?
SSEO writes the content AI assistants quote, builds backlinks and tracks how they talk about you, every month. Try it free for 3 days.