robots.txt for AI crawlers
Use robots.txt to treat AI search crawlers and AI training crawlers separately. To stay visible in AI answers while opting out of training, allow OAI-SearchBot, PerplexityBot and Claude-SearchBot, and disallow GPTBot, ClaudeBot and Google-Extended. Firewall rules must match, and user-initiated fetchers may not follow robots.txt.
Which user agents do what?
| Token | Owner | Purpose |
|---|---|---|
| OAI-SearchBot | OpenAI | ChatGPT search[1] |
| GPTBot | OpenAI | Training[1] |
| PerplexityBot | Perplexity | Search[2] |
| Claude-SearchBot | Anthropic | Search[3] |
| ClaudeBot | Anthropic | Training[3] |
| Google-Extended | Gemini training/grounding[4] |
Template: visible in AI search, no training
User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: /
What robots.txt can’t do
OpenAI says robots.txt rules may not apply to user-initiated ChatGPT-User visits.[1] Perplexity says the same of Perplexity-User.[2] Test with Search Console’s robots.txt report.[5]
Frequently asked questions
Should I block all AI crawlers?
Blocking search crawlers removes you from those engines’ answers.
Does robots.txt stop AI training?
Compliant crawlers honor it; it is not access control.
Where do I put robots.txt?
At the root of each host, e.g. /robots.txt.
See also
References
Sources accessed October 14, 2026 unless another date is given. See all sources.
- “Overview of OpenAI Crawlers”. OpenAI. accessed October 14, 2026.
- “Perplexity Crawlers”. Perplexity. accessed October 14, 2026.
- “Does Anthropic crawl data from the web, and how can site owners block the crawler?”. Anthropic (Claude Help Center). updated April 7, 2026.
- “List of Google's common crawlers (Googlebot, Google-Extended)”. Google for Developers. accessed October 14, 2026.
- “robots.txt report”. Search Console Help. accessed October 14, 2026.