Anthropic · training
ClaudeBot: robots.txt token and blocking it
Everything on this page is taken from Anthropic's own documentation, quoted and linked, and was checked on 4 October 2026. Where the documentation is silent the page says so instead of filling the gap.
What it is
Operator: Anthropic. Kind, as listed here: training.
“ClaudeBot helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training.”
Source: Anthropic: how to block the crawler, checked 4 October 2026.
User-agent string: none published
Anthropic’s page documents the robots.txt token and publishes no full user-agent string for it, so none is shown here.
Source: Anthropic: how to block the crawler. The robots.txt token to match is ClaudeBot.
Does it obey robots.txt?
“Anthropic’s Bots respect “do not crawl” signals by honoring industry standard directives in robots.txt.”
Source: Anthropic: how to block the crawler, checked 4 October 2026.
Does it affect search?
The Anthropic page we checked does not say.
Crawl rate
“To limit crawling activity, we support the non-standard Crawl-delay extension to robots.txt.”
Source: Anthropic: how to block the crawler, checked 4 October 2026.
What else Anthropic says
“When a site restricts ClaudeBot access, it signals that the site's future materials should be excluded from our AI model training datasets.”
Source: Anthropic: how to block the crawler, checked 4 October 2026.
robots.txt snippets
Two snippets that show the syntax for this token. They are examples, not advice: whether to allow or refuse ClaudeBot is your decision, and the effects above are the vendor's description of them.
Allow ClaudeBot
User-agent: ClaudeBot Allow: /
Disallow ClaudeBot
User-agent: ClaudeBot Disallow: /
How to verify requests
Anthropic publishes a list of source IP addresses and says: “If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic.”
https://claude.com/crawling/bots.json
Source: Anthropic: how to block the crawler.
What we measure
ClaudeBot is one of the four agents the Cite Files scanner requests with when it scans a site. Run the AI crawler checker on your own address to see how your site answers it, or read the open record of how scanned sites treat it.
Questions
What is ClaudeBot?
Anthropic describes ClaudeBot like this: “ClaudeBot helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training.”
Does ClaudeBot obey robots.txt?
The Anthropic documentation says: “Anthropic’s Bots respect “do not crawl” signals by honoring industry standard directives in robots.txt.”
How do I block ClaudeBot in robots.txt?
Add a group to robots.txt with the line “User-agent: ClaudeBot” followed by “Disallow: /”. That is an example of the syntax, not a recommendation.
Also see the robots.txt tester for AI crawlers and the other entries in the AI crawler reference.
Last reviewed . Maintained by Cite Files — corrections to [email protected].