SearchVisibility.wikiVisibility in search and AI answers

ClaudeBot

From Search Visibility Wiki · By the · Published · Last reviewed · How we research

ClaudeBot is Anthropic’s web crawler for collecting content that may be used to train its models. Anthropic runs two other agents: Claude-SearchBot, which indexes content to improve Claude’s search results, and Claude-User, which fetches pages when a user asks. All three honor robots.txt, and blocking ClaudeBot does not by itself block the other two.

What are Anthropic’s three bots?

User agentPurposeEffect of blocking
ClaudeBotCollects web content that “could potentially contribute to” model trainingSignals that the site’s future materials should be excluded from training datasets
Claude-SearchBotIndexes content “to improve search result quality”“May reduce your site’s visibility and accuracy in user search results”
Claude-UserFetches pages in response to a user’s questionPrevents retrieval for user queries; “may reduce your site’s visibility for user-directed web search”

Source: Anthropic Help Center, updated April 7, 2026.[1]

How do you block or allow ClaudeBot?

Anthropic says its bots honor robots.txt and asks site owners to add rules for every subdomain they want to opt out:[1]

User-agent: ClaudeBot
Disallow: /

To keep search visibility in Claude while opting out of training, block ClaudeBot but leave Claude-SearchBot and Claude-User allowed. Anthropic also supports the non-standard Crawl-delay directive, for example Crawl-delay: 1.[1]

Why not block Anthropic by IP address?

Anthropic warns that blocking its IP addresses “may not work correctly or persistently guarantee an opt-out,” because it prevents the bots from reading your robots.txt.[1]

How does this compare with GPTBot and Google-Extended?

The pattern is similar elsewhere: OpenAI separates GPTBot (training) from OAI-SearchBot (ChatGPT search),[2] and Google’s Google-Extended token does not affect Google Search.[3] See AI crawlers.

Frequently asked questions

Does ClaudeBot respect robots.txt?

Yes. Anthropic says its bots honor robots.txt directives and do not try to bypass CAPTCHAs.

Will blocking ClaudeBot remove me from Claude’s answers?

Anthropic describes ClaudeBot as a training crawler. Search visibility is tied to Claude-SearchBot and Claude-User, which are controlled separately.

Who do I contact if ClaudeBot misbehaves?

Anthropic lists claudebot@anthropic.com and asks you to write from an address on the affected domain.

References

Sources accessed October 14, 2026 unless another date is given. See all sources.

  1. “Does Anthropic crawl data from the web, and how can site owners block the crawler?”. Anthropic (Claude Help Center). updated April 7, 2026.
  2. “Overview of OpenAI Crawlers”. OpenAI. accessed October 14, 2026.
  3. “List of Google's common crawlers (Googlebot, Google-Extended)”. Google for Developers. accessed October 14, 2026.
Categories: Playbook