Why AI crawlers get blocked when robots.txt allows them
A site can publish a robots.txt that welcomes every AI crawler and still turn those crawlers away, because the two controls sit in different places and neither reads the other. This guide explains the difference, how a scanner can tell, and what that kind of test cannot prove.
Two controls in two places
robots.txt is a file of requests. RFC 9309 describes rules that crawlers are requested to honour, says they are not a form of access authorization, and adds that the protocol is not a substitute for valid content security measures. An edge rule is the other kind of control. A CDN or firewall in front of your site decides whether a request reaches it at all, and that decision does not consult robots.txt.
Edge rules take several forms: a bot-management option, a custom rule matching user-agent text, a rule that refuses traffic from hosting-provider networks, or a security plugin. A setting meant to stop scrapers can catch AI crawlers as well. The request is answered before your application sees it, so it may not appear in the application’s own logs.
What the vendors say
The vendors’ own pages assume the edge matters. OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing requests from its published IP ranges (OpenAI). Perplexity says that if you use a web application firewall you may need to explicitly allow its bots, and its page describes a rule that combines the user-agent with its published IP addresses (Perplexity).
Anthropic says its bots honour robots.txt directives and that “we will not attempt to bypass CAPTCHAs for the sites we crawl”, so a CAPTCHA is a wall for that crawler. It also says blocking the IP addresses its bots use may not persistently guarantee an opt-out, because doing so impedes its ability to read your robots.txt (Anthropic). Google documents Google-Extended as a standalone robots.txt product token with no separate HTTP user agent string (Google), so no request can be made as it and its rule is read from robots.txt alone.
How the two-request comparison works
A scanner cannot read your edge rules. It can only ask. Cite Files requests your homepage as a current desktop browser, then again as each of GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, and compares each answer with the browser’s. The verdict rests on comparing the two responses: the status class, the response size relative to the browser response, and whether your own page title survived. Known challenge-page wording is used only as supporting evidence for a “challenged” verdict, because that wording changes constantly and a keyword test alone is confidently wrong in both directions.
Blocked means an error status where a browser got a page. Challenged means an interstitial, a bot-check page. The crawler requests skip robots.txt, because the question is how the server answers, and robots.txt is read separately by the RFC 9309 rules so the two can be shown side by side. Redirects are followed only within your own site.
If the browser request itself is refused, challenged or unanswered, there is nothing to compare against. The tool then says it cannot judge the site and shows no per-agent result, rather than reporting every crawler as blocked. We request from a data centre, and a rule that refuses hosting-provider networks wholesale can refuse us as well.
What the result cannot prove
The requests come from our address, not from the vendors’ networks. A site that verifies crawlers by their published address will refuse us and still serve the real crawler, so “blocked” means a request with that user-agent from an unknown address was refused. Anthropic documents the ClaudeBot token but not a full string, so ours is the form seen in server logs. In the other direction, a clean result is one request at one moment. It cannot see rules that depend on request volume, geography or reputation, and a site that was fine when we asked can be blocking an hour later.
Settling it with your logs
Logs show what actually arrived. Search your CDN’s logs and your server’s for the crawler names and read the status code beside each: a run of 403 responses next to GPTBot points to an edge block. A user-agent line can be written by anyone, so treat it as a claim and check the address against the ranges the vendor publishes.
If the checker shows a block, the fix is in the CDN or firewall settings, not in robots.txt. Find the rule, allow the crawlers you want to serve, preferably combining the user-agent with the vendor’s published address ranges as Perplexity’s page describes, and run the check again.
Questions
My robots.txt allows GPTBot, so why does the checker say it is refused?
robots.txt is a request that a crawler reads before it fetches. A CDN or firewall rule decides separately, and it can answer with an error or a bot-check page whatever robots.txt says. Look at your bot settings, your custom rules and any rule that refuses hosting-provider networks.
Does a blocked result prove a vendor cannot reach my site?
No. The request comes from our address, not from the vendor's network. A rule that checks the crawler's address could refuse us and still serve the real crawler, so read your access logs to see what the real crawlers receive.
Why does the checker say it cannot judge my site?
The comparison only means something when an ordinary browser request succeeds. If that request was refused, challenged or unanswered, every crawler would appear to match it, so the checker reports no per-agent result rather than guessing.
Check your own site
Each tool is free and needs no account. Run the full check to test crawler access, robots.txt, llms.txt and the rest in one scan, or read how it is scored first.
Last reviewed . Maintained by Cite Files — corrections to [email protected].