Open record · scanner v1.11.0
Who serves AI crawlers, and who turns them away
Every site scanned here is asked for its homepage twice: once as a current desktop browser, once as each crawler. This is what came back.
Not enough sites have been scanned yet to publish anything meaningful. This page fills in as the record grows — 0 sites so far, and we would rather show you nothing than a percentage of five.
How to read this
This sample is not the web. Every site here is one somebody chose to scan, which skews towards sites whose owners suspected a problem. A refusal rate on this page is a fact about our sample and not a measurement of the internet, and we would rather say so than let the number travel without its denominator.
A refusal is not necessarily a mistake. Blocking AI crawlers is a legitimate position. What this page records is what a server actually does, which is frequently not what its owner believes it does — that gap is the reason to publish anything at all.
Each observation is a single request at a single moment from one network. It cannot see rules that depend on volume, geography or reputation. The methodology sets out exactly how a verdict is decided.
This record restarted on 4 October 2026. It counts only scans made with the corrected probe set (scanner 1.11.0 onward), so every figure above describes GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot and nothing older. What changed is in the methodology changelog.
The agent-commerce record asks the next question — not whether a page is reachable, but whether an AI agent can transact with it.
Removal. If your site is named here and you would rather it were not, write to [email protected] from an address at that domain. We remove it, and we do not ask why.