Anthropic · user-triggered
Claude-User: robots.txt token and blocking it
Everything on this page is taken from Anthropic's own documentation, quoted and linked, and was checked on 4 October 2026. Where the documentation is silent the page says so instead of filling the gap.
What it is
Operator: Anthropic. Kind, as listed here: user-triggered.
“Claude-User supports Claude AI users. When individuals ask questions to Claude, it may access websites using a Claude-User agent.”
Source: Anthropic: how to block the crawler, checked 4 October 2026.
User-agent string: none published
Anthropic’s page documents the robots.txt token and publishes no full user-agent string for it, so none is shown here.
Source: Anthropic: how to block the crawler. The robots.txt token to match is Claude-User.
Does it obey robots.txt?
“Anthropic’s Bots respect “do not crawl” signals by honoring industry standard directives in robots.txt.”
Source: Anthropic: how to block the crawler, checked 4 October 2026.
Does it affect search?
“Disabling Claude-User on your site prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search.”
Source: Anthropic: how to block the crawler, checked 4 October 2026.
Crawl rate
“To limit crawling activity, we support the non-standard Crawl-delay extension to robots.txt.”
Source: Anthropic: how to block the crawler, checked 4 October 2026.
robots.txt snippets
Two snippets that show the syntax for this token. They are examples, not advice: whether to allow or refuse Claude-User is your decision, and the effects above are the vendor's description of them.
Allow Claude-User
User-agent: Claude-User Allow: /
Disallow Claude-User
User-agent: Claude-User Disallow: /
How to verify requests
Anthropic publishes a list of source IP addresses and says: “If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic.”
https://claude.com/crawling/bots.json
Source: Anthropic: how to block the crawler.
Questions
What is Claude-User?
Anthropic describes Claude-User like this: “Claude-User supports Claude AI users. When individuals ask questions to Claude, it may access websites using a Claude-User agent.”
Does Claude-User obey robots.txt?
The Anthropic documentation says: “Anthropic’s Bots respect “do not crawl” signals by honoring industry standard directives in robots.txt.”
How do I block Claude-User in robots.txt?
Add a group to robots.txt with the line “User-agent: Claude-User” followed by “Disallow: /”. That is an example of the syntax, not a recommendation. On search, Anthropic says: “Disabling Claude-User on your site prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search.”
Also see the robots.txt tester for AI crawlers and the other entries in the AI crawler reference.
Last reviewed . Maintained by Cite Files — corrections to [email protected].