Which AI crawlers should you allow in robots.txt?

Many sites are ignored by AI answer engines for a boring reason: a robots.txt rule decided it before any content was read. Blocking every "AI bot" with a blanket rule is common and usually a mistake, because the major vendors run several crawlers that do different jobs.

Three kinds of AI crawler

Vendors generally separate their crawlers by purpose. Training crawlers collect pages to build future models. Search crawlers build the index an answer engine uses to find and cite pages. User-triggered fetchers load a page live because a person asked about it.

The main user agents

• OpenAI: GPTBot is for training data. OAI-SearchBot indexes pages for ChatGPT search. ChatGPT-User fetches pages when a user or a custom GPT asks for them.
• Anthropic: ClaudeBot is for training. Claude-SearchBot supports search quality. Claude-User fetches pages on a user's behalf.
• Perplexity: PerplexityBot builds Perplexity's search index. Perplexity-User fetches pages when a user asks a question.
• Google: Googlebot crawls for Search, and AI Overviews and AI Mode draw on that same index. Google-Extended is a robots.txt token that controls whether your content is used for Gemini training and grounding. It does not affect Search ranking.
• Apple and others: Applebot-Extended controls Apple's AI training use. CCBot is Common Crawl, whose dataset many models have been trained on.

What to allow if you want citations

Allow the search and user-triggered agents: OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot and Claude-User, plus Googlebot. These fetch your pages at answer time or build the index answers come from. Blocking them removes you from the pool of pages an engine can cite.

Training crawlers are a business decision. Blocking GPTBot, ClaudeBot or Google-Extended keeps your text out of future training sets and, according to each vendor's documentation, does not remove you from their search products. If you want to opt out of training but stay citable, block only the training tokens.

An example robots.txt

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

Mistakes we see most often

• A blanket User-agent: * with Disallow: / left over from a staging site.
• Blocking at the CDN or firewall instead of in robots.txt. Those rules are invisible in robots.txt, and some CDNs can block AI crawlers by default.
• Never testing the file. A rule that lands in the wrong user-agent group silently applies the wrong policy.

Vendor documentation changes, so check each company's current crawler page before relying on a user-agent name. You can run the free Cited Lens audit to see which of these agents your robots.txt currently blocks.