robots.txt tester for AI crawlers

How a crawler picks its group and its rule

A robots.txt is made of groups: one or more User-agent lines followed by Allow and Disallow rules. Under RFC 9309, a crawler finds the group whose user-agent line most specifically matches its own name and uses User-agent: * only when no group does. Inside that one group, the longest matching path rule decides, and Allow wins when two rules are the same length. This tester follows that order. Like the full scan, it matches a user-agent line as a case-insensitive part of the crawler’s name, and its rules may use * and a closing $.

Why the * rules do not apply to an agent with its own group

Groups are not layered. Once a crawler has a group of its own, the * group is not consulted for it at all:

User-agent: *
Disallow: /private

User-agent: GPTBot
Allow: /

Here GPTBot is allowed at /private, because the only group that applies to it has no rule against that path. Every other agent in the list is disallowed there. This is the most common way a file does the opposite of what its author meant.

One company, several tokens

Vendors publish more than one user-agent token, each for a different job: gathering content for model training, building a search index, or fetching a page because a person asked an assistant about it. A rule for one token does not govern another unless the file says so. Which token does which job is stated by the vendor, not by this page: see OpenAI, Anthropic, Perplexity and Google. The tester evaluates 27 tokens and lists the 11 the scanner treats as critical first.

What this tool does and does not do

Pasted text is evaluated and never fetched. With an address, it requests /robots.txt on that site as CiteFilesBot, and nothing else: one request, or up to three if your own site redirects it, for example from http to https or to the www address. A redirect to another site is reported and not followed, and the check gives up after 12 seconds. The path you enter is compared with the rules and is never requested. It reads User-agent, Allow, Disallow, Sitemap and the Content-Signal line, quotes the last two, and does not validate other directives.

It does not read your robots.txt before fetching it, which would be circular. It asks for the one named file directly, because robots.txt is a file published for crawlers to read. The llms.txt generator and the full scan do honour robots.txt before reading pages.

It does not test whether a crawler is actually served your pages. robots.txt is a request, not an enforcement, and an edge rule can refuse a crawler the file allows; the AI crawler access checker compares the two. When the same agent is named in two groups, RFC 9309 combines them but this tester reads only the first, so it flags the file for a manual check.

Reading the result

“No rule matched” means nothing in the group covers that path, so the agent is allowed. A file that answers 200 with an HTML page is not a robots.txt, so no verdicts are shown; the same goes for a server error or a refused request. A file with no robots.txt at all is shown as every agent allowed.

Questions