Skip to content
Cite Files

Open record · scanner v1.14.0

Who serves AI crawlers, and who turns them away

Every site scanned here is asked for its homepage twice: once as a current desktop browser, once as each crawler. This is what came back.

CrawlerServedTurned awayRefusal rate
ClaudeBot1200%
GPTBot1200%
OAI-SearchBot1200%
PerplexityBot1200%

One row per site — the most recent scan of each — across 12 sites.

How to read this

Two populations, never mixed. From 3 November 2026 we will also measure a published sample: a fixed, dated list of well-known sites, published here when the first edition starts, measured on a schedule after each site’s robots.txt allows it. Today the record covers only sites somebody chose to scan, which skews towards sites whose owners suspected a problem. A refusal rate on this page is a fact about that population and not a measurement of the internet, and we would rather say so than let the number travel without its denominator.

A refusal is not necessarily a mistake. Blocking AI crawlers is a legitimate position. What this page records is what a server actually does, which is frequently not what its owner believes it does — that gap is the reason to publish anything at all. Every crawler request is sent from our servers, not the vendors’ networks, so a site that checks where a request comes from may refuse us and still serve the real crawler.

Each result is one request per crawler from one of our servers. The published sample will bring a second server on a different network address that repeats it; neither can see rules that depend on volume, geography or the vendor’s own network. The methodology sets out exactly how a verdict is decided.

This record restarted on 4 October 2026. It counts only scans made with the corrected probe set (scanner 1.11.0 onward), so every figure above describes GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot and nothing older. What changed is in the methodology changelog.

The agent-commerce record asks the next question — not whether a page is reachable, but whether an AI agent can transact with it.

No site is named on this page yet. The aggregate is published; individual hosts are not. When we do start naming sites, only sites in the published sample will be named, and only when two of our servers on different network addresses got the same answer within three days.

Removal. Disallow CiteFilesBot in robots.txt (for sites in the published sample it will be re-checked at least every seven days), or write to [email protected] from an address at that domain. We remove it within two working days and do not ask why. Sites people chose to scan are counted, never named.