AI crawlers explained: who to allow, who to block

“Should we block AI crawlers?” has become a standard question, and the standard answer, a blanket block on everything with AI in its name, quietly destroys visibility. The confusion comes from treating AI crawlers as one group. They’re two groups with opposite consequences, and a good robots.txt treats them differently.

The distinction that matters: search vs. training

Search (or answer) crawlers fetch your pages so an assistant can cite and recommend you in a live answer. They’re the AI equivalent of Googlebot, and blocking them makes you invisible in AI answers. Training crawlers collect content to train future models. They have nothing to do with whether you show up, and blocking them is a legitimate content-policy choice that costs you exactly zero visibility. Every robots.txt decision gets easy once you sort the bot into the right group.

The search crawlers: let these through

Four matter most right now. OAI-SearchBot indexes pages for ChatGPT’s search feature. ChatGPT-User fetches a page live when a user’s question calls for it. PerplexityBot powers Perplexity’s cited answers. Claude-SearchBot does the same for Claude. Disallow any of these and that assistant can’t read your site, which means it can’t name you when someone nearby asks for exactly what you sell. For a local business there is no upside to blocking them.

The training crawlers: your call

GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google) and CCBot (Common Crawl) collect content for model training. Allowing or blocking them is a philosophy-and-business decision, not a visibility one. Google-Extended is the cleanest example: blocking it opts your content out of training Google’s AI models and has no effect on your Google Search rankings or your appearance in AI Overviews.

Copy-ready robots.txt rules, explained

Each rule is two plain lines in your robots.txt. Here’s what each one does:

• User-agent: GPTBot → Disallow: / (blocks OpenAI model training only; ChatGPT can still cite you via OAI-SearchBot).
• User-agent: CCBot → Disallow: / (blocks the Common Crawl dataset many models train on).
• User-agent: Google-Extended → Disallow: / (opts out of Google AI training; normal search is untouched).
• User-agent: OAI-SearchBot → Allow: / (keeps you visible in ChatGPT search answers).
• User-agent: PerplexityBot → Allow: / (keeps you citable in Perplexity).
• User-agent: Claude-SearchBot → Allow: / (keeps you citable in Claude).

The blanket-block mistake

The worst pattern in the wild: a security plugin or a well-meaning agency adds a wall of Disallow rules for every known AI user agent, search crawlers included. The site owner believes their content is protected; in reality they’ve removed themselves from every AI answer while their competitors stay in. If someone set up “AI blocking” on your site, check exactly which user agents it covers before assuming it was the right call.

A sane default for a local business

Allow the four search crawlers. Decide consciously about the training crawlers. For most local businesses the content isn’t sensitive enough to justify blocking, but either choice is defensible. And re-check after every website relaunch or plugin update: robots.txt regressions are silent, and you won’t notice you’ve vanished from AI answers until the calls stop coming.

Check your site in 10 seconds

The free ScanToPitch AI Visibility Check at scantopitch.com/audit reads your robots.txt and shows you instantly which AI crawlers you allow and which you block (search and training separated). No signup required.