SearchVisibility.wikiVisibility in search and AI answers

AI crawlers and robots.txt

From Search Visibility Wiki · By the · Published · Last reviewed · How we research

To be visible in AI answers you must let each engine’s search crawler in: Googlebot (Google Search including AI Overviews and AI Mode), OAI-SearchBot (ChatGPT search) and PerplexityBot (Perplexity). Separate tokens such as GPTBot and Google-Extended control model training and some grounding uses; blocking them does not remove you from Google Search or ChatGPT search.

Which crawler controls what?

TokenControlsSource
GooglebotGoogle Search, including all Search features[1]
Google-ExtendedGemini training; grounding in Gemini Apps and Vertex AI; not Search[1]
OAI-SearchBotChatGPT search inclusion[2]
GPTBotOpenAI training[2]
PerplexityBotPerplexity search inclusion (not training)[3]

What does a visibility-first robots.txt look like?

User-agent: Googlebot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Optional: opt out of model training only
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Robots.txt is a request, not access control. User-initiated fetchers (ChatGPT-User, Perplexity-User) may not follow it.[2][3]

Why do CDN and firewall rules matter?

Google lists “ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure” among its SEO fundamentals for AI features.[4] OpenAI and Perplexity both recommend allowing their published IP ranges.[2][3]

Frequently asked questions

Does llms.txt replace robots.txt?

No. llms.txt is an unofficial proposal; Google says Google Search ignores it. Robots.txt remains the crawl control.

Can I block AI Overviews with robots.txt?

Only by blocking Googlebot, which removes you from Search. Google suggests snippet controls such as nosnippet to limit what is shown.

How long do robots.txt changes take?

OpenAI says about 24 hours for ChatGPT search; Perplexity says up to 24 hours.

References

Sources accessed October 14, 2026 unless another date is given. See all sources.

  1. “List of Google's common crawlers (Googlebot, Google-Extended)”. Google for Developers. accessed October 14, 2026.
  2. “Overview of OpenAI Crawlers”. OpenAI. accessed October 14, 2026.
  3. “Perplexity Crawlers”. Perplexity. accessed October 14, 2026.
  4. “AI features and your website”. Google Search Central. accessed October 14, 2026.
Categories: Engines