Anthropic · search
Claude-SearchBot: robots.txt token and blocking it
Everything on this page is taken from Anthropic's own documentation, quoted and linked, and was checked on 4 October 2026. Where the documentation is silent the page says so instead of filling the gap.
What it is
Operator: Anthropic. Kind, as listed here: search.
“Claude-SearchBot navigates the web to improve search result quality for users. It analyzes online content specifically to enhance the relevance and accuracy of search responses.”
Source: Anthropic: how to block the crawler, checked 4 October 2026.
User-agent string: none published
Anthropic’s page documents the robots.txt token and publishes no full user-agent string for it, so none is shown here.
Source: Anthropic: how to block the crawler. The robots.txt token to match is Claude-SearchBot.
Does it obey robots.txt?
“Anthropic’s Bots respect “do not crawl” signals by honoring industry standard directives in robots.txt.”
Source: Anthropic: how to block the crawler, checked 4 October 2026.
Does it affect search?
“Disabling Claude-SearchBot on your site prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results.”
Source: Anthropic: how to block the crawler, checked 4 October 2026.
Crawl rate
“To limit crawling activity, we support the non-standard Crawl-delay extension to robots.txt.”
Source: Anthropic: how to block the crawler, checked 4 October 2026.
robots.txt snippets
Two snippets that show the syntax for this token. They are examples, not advice: whether to allow or refuse Claude-SearchBot is your decision, and the effects above are the vendor's description of them.
Allow Claude-SearchBot
User-agent: Claude-SearchBot Allow: /
Disallow Claude-SearchBot
User-agent: Claude-SearchBot Disallow: /
How to verify requests
Anthropic publishes a list of source IP addresses and says: “If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic.”
https://claude.com/crawling/bots.json
Source: Anthropic: how to block the crawler.
Questions
What is Claude-SearchBot?
Anthropic describes Claude-SearchBot like this: “Claude-SearchBot navigates the web to improve search result quality for users. It analyzes online content specifically to enhance the relevance and accuracy of search responses.”
Does Claude-SearchBot obey robots.txt?
The Anthropic documentation says: “Anthropic’s Bots respect “do not crawl” signals by honoring industry standard directives in robots.txt.”
How do I block Claude-SearchBot in robots.txt?
Add a group to robots.txt with the line “User-agent: Claude-SearchBot” followed by “Disallow: /”. That is an example of the syntax, not a recommendation. On search, Anthropic says: “Disabling Claude-SearchBot on your site prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results.”
Also see the robots.txt tester for AI crawlers and the other entries in the AI crawler reference.
Last reviewed . Maintained by Cite Files — corrections to [email protected].