Open record · scanner v1.2.0
Who serves AI crawlers, and who turns them away
Every site scanned here is asked for its homepage twice: once as a current desktop browser, once as each crawler. This is what came back.
| Crawler | Served | Turned away | Refusal rate |
|---|---|---|---|
| ClaudeBot | 10 | 3 | 23% |
| GPTBot | 12 | 1 | 8% |
| PerplexityBot | 12 | 1 | 8% |
| Google-Extended | 13 | 0 | 0% |
One row per site — the most recent scan of each — across 13 sites.
How to read this
This sample is not the web. Every site here is one somebody chose to scan, which skews towards sites whose owners suspected a problem. A refusal rate on this page is a fact about our sample and not a measurement of the internet, and we would rather say so than let the number travel without its denominator.
A refusal is not necessarily a mistake. Blocking AI crawlers is a legitimate position. What this page records is what a server actually does, which is frequently not what its owner believes it does — that gap is the reason to publish anything at all.
Each observation is a single request at a single moment from one network. It cannot see rules that depend on volume, geography or reputation. The methodology sets out exactly how a verdict is decided.
No site is named on this page. The aggregate is published; individual hosts are not, because somebody scanning their own site privately did not sign up to be listed.