AI Crawler Reference 2026

Opt out of AI training and indexing

Add this block to your robots.txt at the root of your domain (e.g., https://example.com/robots.txt). It tells every reputable AI crawler not to fetch any URL on your site:

# Block AI crawlers — generated by sitemaphost.app/ai-crawlers/
# Last reviewed: Mon May 18 2026 02:00:00 GMT+0200 (Central European Summer Time)

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Bytespider
User-agent: CCBot
User-agent: cohere-ai
User-agent: Diffbot
User-agent: DuckAssistBot
Disallow: /

Caveats: Google-Extended and Applebot-Extended are opt-out tokens, not separate fetchers. They tell the main Googlebot / Applebot crawlers that your content is off-limits for AI training, while still letting Search index you. If you want to opt out of search indexing too, also block Googlebot and Applebot — but be aware this removes you from Google and Apple Search results.

Block only training, keep RAG citations

If you want ChatGPT, Claude, and Perplexity to cite your pages in live answers (and drive traffic) but don't want them training models on you, block only the training-corpus crawlers:

# Block training crawlers; allow live-citation crawlers

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: meta-externalagent
User-agent: Bytespider
User-agent: CCBot
User-agent: cohere-ai
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# These continue to fetch (and cite) your pages:
# OAI-SearchBot, ChatGPT-User, Claude-User, Claude-SearchBot,
# PerplexityBot, Perplexity-User, DuckAssistBot, Diffbot

This is the recommendation for most B2B SaaS, marketing sites, and content businesses — AI citations are a fast-growing traffic source (see Plausible-tracked sources for sitemaphost.app itself) and blocking RAG crawlers cuts you off from that. Only training-corpus blocking matters for the "I don't want to be training data" goal.

How to detect AI crawler traffic

If you want to measure rather than block, log requests by User-Agent header and bucket them by crawler family. The user-agents in the table above are case-sensitive on their distinguishing token; matching with a case-insensitive substring is the safest approach.

A simple pseudo-rule in nginx / Cloudflare Workers / your access-log pipeline:

const ai = /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|meta-externalagent|Bytespider|CCBot|cohere-ai|Diffbot|DuckAssistBot/i;
if (ai.test(request.headers.get('user-agent') || '')) {
  // log to analytics, add 'ai-crawler' header, etc.
}

For a higher-confidence signal you can also verify the source IP against the operator's published ranges (linked in the table). OpenAI and Anthropic publish JSON files; Google-Extended uses Google's main ranges; some operators don't publish IP lists at all, in which case the user-agent string is the only signal.