How the score works

We request your homepage as a current desktop browser, then again as GPTBot, ClaudeBot, PerplexityBot and Google-Extended, using their real published user-agent strings. Then we compare the four answers against the browser one.

A verdict of blocked means an error status where a browser got a page. Challenged means an interstitial: a bot-check page, which crawlers do not solve. We decide that structurally — status class, response size relative to the browser response, and whether your own <title> survived — rather than by looking for vendor phrases, because block-page wording changes constantly and a keyword test produces confident nonsense in both directions.

This is a single request per crawler from one address at one moment. It cannot see rules that depend on request volume, geography or reputation, and a site that was fine when we asked can be blocking an hour later. Treat a clean result as evidence, not a guarantee.

Read the way RFC 9309 says a crawler reads it: the most specific matching user-agent group wins, * is the fallback, the longest matching rule decides, and Allow beats Disallow on equal length. A file containing the word “Disallow” is not necessarily blocking anyone, and a file that looks welcoming can still block everything.

We evaluate the root path for 27 crawler tokens. We also detect Cloudflare’s managed robots.txt, which is injected above your own file and disallows the AI crawlers — so the rules you wrote are never reached.

Many sites answer 200 OK with their HTML shell for any path they do not recognise. A checker that treats 200 as “found” will report an llms.txt on a site that has never had one.

So before anything else we request a deliberately impossible path. If that returns 200, every subsequent 200 is compared against the shell it returned. A file only counts as present when the body is a file rather than a page.

Paths we check:

• /llms.txt
• /llms-full.txt
• /.well-known/llms.txt
• /soul.md
• /memory.md
• /AGENTS.md
• /ai.txt
• /.well-known/ai-plugin.json
• /.well-known/mcp.json
• /openapi.json
• /.well-known/security.txt
• /humans.txt

Almost everyone conflates two separate questions, so we keep them apart. Retrieval is whether an assistant may fetch your page to answer a question now, and cite you — governed by robots.txt and by whether your edge actually serves the crawler. That is what most of this report is about. Training is whether your content may become weights in a model. That is a licensing question, and ai.txt is where it is declared.

Wanting the first and not the second is a perfectly coherent position, and a common one. Without a declaration nobody can tell which you meant, and silence tends to be read as whatever suits the reader.

Two caveats we print every time: ai.txt is a convention rather than a ratified standard, and adoption is partial. It records intent, it does not enforce anything. It matters because a stated reservation is evidence of intent and an unstated one is not.

We read it with the same RFC 9309 grammar as robots.txt, and flag the contradiction that actually matters: granting training use while disallowing the crawlers from reading the site at all. The reverse — allowing retrieval and reserving training — is coherent and we leave it alone. The ai.txt we generate is derived from the position your robots.txt already expresses; we do not hold a view on how you should licence your work.

We recommend declaring Content-Signal in robots.txt, and Lighthouse marks any robots.txt containing it as invalid. Both of those are true at once, so rather than pick a side we measured it: removing that single line takes this site’s Lighthouse SEO score from 92 to 100 and changes nothing about what a crawler does.

Lighthouse’s parser predates the directive. We keep the line and accept the eight points, because the declaration is the point and an SEO sub-score is a proxy. If you follow the advice and your number moves, that is why — it is flagged in your technical report rather than left for you to discover.

We give a language model only the text a crawler receives from your pages — no search, no browsing, no outside knowledge — and ask it seven questions a customer would ask an assistant about you.

An answer only counts if the model returns a verbatim quote from your text, and we then check that quote really appears in what we sent. If it does not, the model was filling a gap rather than reading one, and the question is marked unanswerable. Those discarded answers are counted and shown to you, because that is precisely what happens when a real assistant is asked about you and your pages come up short.

The answerability test asks fixed questions. This asks the opposite: when a model is asked about you and your pages come up short, what does it say instead?

One model is given your page text and asked to describe your organisation the way it would to someone who asked — with no instruction to stay grounded, because that is how a real assistant behaves. A second, independent model then has to find a verbatim quote supporting each statement, and we check that quote actually occurs in the text we supplied. Statements that survive neither gate are the shape of the thing an assistant will tell a customer that your site never said.

This is a measure of your content’s ambiguity, not of the model’s honesty. Where your pages are specific, models stay accurate.

Sector drives two things: which schema.org types we recommend, and which peer group your score is compared against. It is decided by rules, not by a model, and the report shows the signals that produced it — a classification you cannot audit is not worth acting on.

Two kinds of signal, and the difference matters. Presence signals are settled by a single page: a checkout page means a shop however small, a LegalService entity means a practice. Share signals only mean something in proportion — nearly every company has a blog, so having one says nothing, while two thirds of the site being articles says publisher. Counting both the same way is how a payments company gets classified as a newspaper, which is exactly what happened before this distinction existed.

When the evidence points two ways at once we say “general” rather than pick. The recommended schema list for a sector is fixed, so two competitors get identical advice and neither gets a type invented for them.

Your score against the sites we have scanned in your sector: the median, the middle half, and where you sit. One scan per site — the most recent — so a site scanned forty times during a fix session cannot dominate its own sector.

Below eight comparable sites we widen to every sector; below eight again we show nothing, because a median of three is not a benchmark.

The caveat is the important part. Our sample is people who came here because they suspected a problem, so it skews low. Beating this median means you are ahead of sites whose owners were already worried — not ahead of the web. Treat it as a floor.

You can ask us to re-check a site once a week. We email you only when something material moved: the score shifted by two or more, a crawler started being turned away, a file appeared or vanished, or a new serious problem exists. Cosmetic churn does not earn an email, because a monitoring message that usually says nothing gets filtered — and then the one that matters gets filtered with it.

Every scan records which build of the scanner produced it. When we improve how we measure, comparisons across that change are labelled as ours rather than presented as your site’s movement. This exists because of a real false alarm: a sampling improvement moved a site by six points and the watch told its owner their site had improved. It had not — we had.

A watch that fails backs off, and gives up after five consecutive failures rather than re-scanning an abandoned domain every week forever.

Every measured check on your report is arithmetic on things we fetched. No score is ever set by a model.

The model writes the commentary, and it compresses your own page text into descriptions for your llms.txt. It is never asked what your business does, because it would answer. Where a generated file needs a fact we have no source for — your social profiles, your licence, your correction policy — you get a visible TODO instead of a plausible invention.

If the model is unavailable, the report still arrives: you get the measured half in full and a note saying the commentary was skipped.

We read your homepage plus up to twenty-four more pages, chosen from your sitemap where you have one and from your homepage links where you do not. Selection favours the pages that describe the business — about, services, pricing, docs, contact — and caps any one section at three pages, so a large blog cannot crowd out everything that matters. A site’s nine-hundredth post tells us nothing the third did not.

This means structure and schema figures are a sample, not a census. Where a category is scored on a proportion, the report says how many pages that proportion is out of.

Requests identify themselves as CiteFilesBot except in the crawler-access test, where the point is to send a specific agent string. We read one page at a time per site with a small amount of parallelism, cap every response, and stop after a fixed budget. We only ever request public pages over HTTP and HTTPS on the standard ports. More about our crawler.