Common Crawl · archive
CCBot: user agent and how to block it
Everything on this page is taken from Common Crawl's own documentation, quoted and linked, and was checked on 4 October 2026. Where the documentation is silent the page says so instead of filling the gap.
What it is
Operator: Common Crawl. Kind, as listed here: archive.
“Common Crawl is a non-profit foundation founded with the goal of democratizing access to web information by producing and maintaining an open repository of web crawl data that is universally accessible and analyzable by anyone.”
Source: Common Crawl: CCBot, checked 4 October 2026.
User-agent string
CCBot/2.0 (https://commoncrawl.org/faq/)
Source: Common Crawl: CCBot. The robots.txt token to match is CCBot.
Does it obey robots.txt?
“To prevent Common Crawl from crawling your website, include the following in your robots.txt”
Source: Common Crawl: CCBot, checked 4 October 2026.
Does it affect search?
The Common Crawl page we checked does not say.
What else Common Crawl says
“Please note that we are aware of crawlers falsely identifying themselves as CCBot. We recommend verifying UserAgent strings to ensure authenticity.”
Source: Common Crawl: CCBot, checked 4 October 2026.
“CCBot is now run on dedicated IP address ranges with reverse DNS (except over IPv6 where reverse DNS is not yet supported.)”
Source: Common Crawl: CCBot, checked 4 October 2026.
robots.txt snippets
Two snippets that show the syntax for this token. They are examples, not advice: whether to allow or refuse CCBot is your decision, and the effects above are the vendor's description of them.
Allow CCBot
User-agent: CCBot Allow: /
Disallow CCBot
User-agent: CCBot Disallow: /
How to verify requests
Common Crawl says CCBot runs on dedicated IP address ranges with reverse DNS (except over IPv6, where reverse DNS is not yet supported), and publishes the ranges as JSON.
https://index.commoncrawl.org/ccbot.json
Source: Common Crawl: CCBot.
Questions
What is CCBot?
Common Crawl describes CCBot like this: “Common Crawl is a non-profit foundation founded with the goal of democratizing access to web information by producing and maintaining an open repository of web crawl data that is universally accessible and analyzable by anyone.”
Does CCBot obey robots.txt?
The Common Crawl documentation says: “To prevent Common Crawl from crawling your website, include the following in your robots.txt”
How do I block CCBot in robots.txt?
Add a group to robots.txt with the line “User-agent: CCBot” followed by “Disallow: /”. That is an example of the syntax, not a recommendation.
Also see the robots.txt tester for AI crawlers and the other entries in the AI crawler reference.
Last reviewed . Maintained by Cite Files — corrections to [email protected].