Rubric v1.2.0 · ACR rubric v1.1.0 · scanner v1.11.0
What has changed in the scoring
A versioned score is only worth anything if you can find out what the versions mean. Otherwise “your score went up” is a claim nobody can check.
Worth saying plainly: several of the changes below were found by running our own tools on our own site, and each of them happened to raise our score. We think each is defensible on its merits — the reasoning is in the entry — but “trust us” is not something you should accept from the party doing the scoring, so here is the record instead. Every entry says which way it moved scores.
- scanner v1.11.0
We probed a user agent that does not exist
The crawler-access measurement requests a site's homepage as a browser and then under four crawler names, and we described the four user-agent strings we sent as real, published crawler strings. That was wrong in four ways. Google-Extended has no user-agent string at all: Google documents it as a robots.txt token that controls whether content may be used for Gemini training and grounding, and says it has no separate HTTP request user agent. So the string we sent under that name was one we made up, no real request ever looks like it, and the result said nothing about Google. Our GPTBot string carried version 1.2 where OpenAI documents 1.4. Our PerplexityBot string had its parenthesis in a different place from the string Perplexity documents. And Anthropic documents the ClaudeBot token but publishes no full user-agent string, so ours could not be called published. From scanner 1.11.0 the four agents are GPTBot and OAI-SearchBot, sent with the strings OpenAI documents; ClaudeBot, sent in the form seen in server logs, which we now describe as exactly that and not as a vendor string; and PerplexityBot, sent with the string Perplexity documents. OAI-SearchBot replaces Google-Extended because it is the crawler that decides whether a site can appear in ChatGPT search. Google-Extended is still read where it belongs, as a token in robots.txt: the robots.txt checks and the files we generate are unchanged. The Agent Commerce Readiness price comparison sends the same agents, so it moves to the same four.
Effect on scoresResults for Google-Extended in earlier reports measured nothing about Google and should be disregarded; older reports now say so beside the table. Scores may move by a small amount for sites that treated the invented string differently from a browser. They may also move for a site that treats any of the other agents differently from the old strings, because OAI-SearchBot is probed for the first time and the GPTBot, ClaudeBot and PerplexityBot strings changed. The edge-access points are still shared equally across the agents probed, there are still four, and no weight changed, so the rubric version stays 1.2.0. The public record of how sites treat AI crawlers counts only scans made with the corrected probe set, so it restarts. Weekly watch emails and before-and-after comparisons now compare only the agents both scans asked as, and will carry the scanner-changed caveat.
Sources: Google common crawlers · OpenAI: crawlers and user agents · Perplexity crawlers · Anthropic: how to block the crawler
- ACR rubric v1.1.0scanner v1.10.0
Agent-payable: a sixth rung
Agent Commerce Readiness gains a sixth rung, T5 Agent-payable, above T4 — the ladder now runs T0 to T5: the site answers with a machine-payable challenge — x402 v1, x402 v2 or L402 — that an agent holding credentials could settle without a human. The predicates for T0 through T4 are unchanged. The 402 family, previously recorded only as a watch item, now moves onto the graded ladder when the challenge is x402 or L402; Cloudflare's pay-per-crawl toll and a bare 402 with no machine-readable challenge stay recorded only, because a crawler toll and a WAF's payment-signalling are not a commerce offer. The ladder stays strictly sequential: T5 is only reached after T4 is proven met.
Effect on scoresNo Citation Readiness score moves. No existing Agent Commerce Readiness tier moves: T5 sits above T4 and nothing below it changed. The public record counts only audits under the current ACR rubric, so it restarts from zero until sites are re-audited; nobody was named under 1.0.0.
Sources: x402 payment protocol · L402: Lightning HTTP 402 protocol
- scanner v1.10.0
Finding the commerce page
Agent Commerce Readiness's price-page classifier only recognised a handful of English path names — pricing, plans, price — so a site whose commerce page lives at /pro, /premium, /upgrade, /subscribe, /buy, /checkout, or a localised name such as /preise, /tarifs or /precios was audited as though it had no price page to look at, reading Ungraded no matter what price a browser could actually see there. The classifier now recognises those paths too, matched as a whole final path segment so that a /pro section of a blog post is not mistaken for a pricing page. When no page is classified pricing or offering at all, the audit now falls back to the sampled page whose extracted text carries the most price-shaped tokens rather than giving up outright; a site owner can also declare the URL of their own commerce page directly on the report, which the audit treats as a same-origin, robots-allowed pricing page and records as owner-declared wherever the result is shown. Separately, the audit no longer re-fetches .well-known/mcp.json, openapi.json and ai-plugin.json when the Citation Readiness scan that runs alongside it already fetched them a few seconds earlier — it reuses that result instead.
Effect on scoresNo Citation Readiness score moves. Agent Commerce Readiness audits of sites whose pricing page lives at a path we did not recognise may now read T0 instead of Ungraded — including our own, whose /pro page carries no price yet. Fewer requests per audit for sites that publish a well-known manifest.
- ACR rubric v1.0.0scanner v1.9.0
Agent Commerce Readiness: a second question, answered without a score
A new audit, run separately from the Citation Readiness scan and only when a site owner asks for it: the Agent Commerce Readiness matrix. Eight read-only checks — whether an agent gets served the same commerce page and price as a browser, whether prices are text a machine can read without running the page, whether an unauthenticated API request tells an agent how to behave, whether robots.txt agrees with what the edge actually enforces, whether an agent can obtain its own credentials, and whether a real machine interface exists (OpenAPI, MCP, or a UCP manifest) — are bundled by published predicate into one of five tiers, T0 to T4 (a sixth tier, T5, was added in ACR rubric 1.1.0 — see the entry above), or left ungraded when the browser baseline itself could not be read. Four watch items (the 402 payment family, RFC 9728, agents.json, ai-plugin.json) are recorded but never graded. It sends never more than forty read-only requests, usually well under, never a form and never a payment, and honours the site's robots.txt for every path. Before building it, four assumptions were checked against the live specs and corrected: MCP's handshake turned out to need a GET-first 405 fingerprint rather than a blind POST, x402's challenge lives in a header format neither v1 nor v2 agree on precisely, the rate-limit field names in the current IETF draft are not what most APIs actually ship, and RFC 8414's default grant type is authorization_code plus implicit — so an authorisation server that says nothing about client_credentials is declaring it absent, not leaving it unknown.
Effect on scoresNo Citation Readiness score moves anywhere, for any site, including ours. This is a separate audit with its own version, and it produces a tier, not a number — a number we could not defend this month is worse than no number.
Sources: RFC 9309: Robots Exclusion Protocol · OpenAPI Specification 3.1.0 · Model Context Protocol specification · Universal Commerce Protocol · RFC 9728: OAuth 2.0 Protected Resource Metadata · x402 payment protocol · IETF draft: RateLimit header fields for HTTP · RFC 8414: OAuth 2.0 Authorization Server Metadata
- scanner v1.8.0
A domain corroboration panel — outside evidence, not a score
A new panel on the full report asks four free, outside sources what they know about the domain that everything else in this audit cannot ask about itself: when the Wayback Machine first archived it, its public RDAP registration date, whether it appears in the Tranco ranking of the web's most-requested domains, and whether Google Safe Browsing currently flags it. Each of the four distinguishes three outcomes rather than two — a value, a genuine "no record" (never archived, no public registration date, not ranked), and "not measured" (the lookup failed, timed out, or was never configured) — so an absence never reads as a finding when we simply failed to ask. Safe Browsing carries an extra guard: every call bundles a URL Google is known to flag alongside the domain's, and a clean result is only reported once that control URL comes back flagged in the same batch, otherwise the honest answer is that the check could not be verified. Absence from Tranco or a young registration date is normal for most legitimate sites and is presented as such, not as a criticism.
Effect on scoresNo score moves anywhere, for any site, including ours. This panel is entirely unscored — every one of these four checks is context for a human reader, not an input to the rubric, and the panel says so on its face. The only thing that changes is what a report shows underneath the existing checks.
Sources: Internet Archive Wayback Machine · RFC 9083: RDAP JSON responses · Tranco list · Google Safe Browsing
- scanner v1.7.0
Very large sitemaps no longer kill the scan
A site whose sitemap declared more than about 125,000 URLs crashed our scanner outright — no score, no report, just a failure. The cause was ours and unglamorous: we merged sitemap entries by spreading one list into another, and JavaScript limits how many items that can carry. A sitemap index with five children of 50,000 URLs each is entirely normal for a large news or retail site, and is well past that limit. We now merge without spreading, and analyse the first 100,000 URLs, which is above anything that scanned successfully before.
Effect on scoresSites that already scanned are completely unaffected — the limit sits above every size that used to work. Sites that used to fail now get a real report, based on a sample of their sitemap, and the report says on its face that it is a sample and how many URLs were left out. No score anywhere moves because of this change; scores appear where previously there was nothing at all.
- scanner v1.6.2
The Wikidata check had never once worked
We match a site to its Wikidata entry through the "official website" property, which is how we can be sure an entry is yours and not a similarly-named organisation's. The query asked for that website as a quoted piece of text, while Wikidata stores it as a web address — two different kinds of thing, which can never match. The lookup therefore returned nothing for every site, every time, since the day it shipped. It now asks in the form that can match: verified against four organisations that plainly do have entries — Stripe, the BBC, Monzo and SQLite — none of which the old query could find and all of which the new one finds.
Effect on scoresNobody's score changes; this check has never been scored. What changes is that sites with a Wikidata entry will now be told so, and told whether their structured data links to it. Every report issued before today said, in effect, "no entry found" — please disregard that, it was not a finding about your site. A check that can only ever return one answer looks exactly like a working check, which is why this survived so long, and why the test that now guards it asserts the shape of the question rather than the answer.
Sources: Wikidata
- scanner v1.6.1
Entity, rendering and retrieval findings now shown, not just measured
The entity-corroboration, JavaScript-rendering and search-retrieval audits have always run and their results have always been stored on your report. But the code that turned each audit's result into a written finding was never called from the report itself, so nothing from any of the three ever reached the page — Wikidata matches, invisible JavaScript-only pages, absent search results, all measured and then silently discarded. They are now derived from the same stored data and merged into your list of findings.
Effect on scoresNo score changes at all: this is not a scoring fix, and the arithmetic behind your number is untouched. What changes is that findings which were already measured, on the report you already have, now appear in the issue list when you next open it. This was our bug, not yours, and it means reports issued before today will show more findings than they did on first read — not because your site changed, but because we are finally telling you what we already knew.
Sources: Wikidata
- scanner v1.6.0
Schema recommendations written as alternatives are now credited
Five of our sector recommendations offer a choice — "ProfessionalService or LegalService", "NewsArticle or Article", "MedicalOrganization or Physician", "Menu or OfferCatalog", "Article or WebPage". The check that decided whether you had published one was meant to split those on the word "or" and accept either. A corrupted character in the pattern meant it never split them, so it compared your published types against the whole phrase, which matches nothing. Every site publishing one of those types was marked as missing it.
Effect on scoresRaises scores for legal, medical, news, hospitality and general-content sites that had already done the right thing and were being told they had not. No site scores lower. This was our bug for its whole life, and it only ever marked sites down.
Sources: Schema.org
- scanner v1.5.0
Word boundaries, paragraphs, and navigation excluded from your content
Our HTML-to-text step joined adjacent elements with nothing between them, so a navigation bar came out as "Skip to contentHomeShopBlogAbout" — one word, as far as the word count was concerned. It also kept the header and footer as part of every page's content, and then flattened what was left into a single paragraph. The text is now extracted with proper word and paragraph boundaries, and site furniture is excluded twice over: structurally, and by removing lines that repeat across most of your pages.
Effect on scoresWord counts change in both directions — up where words were glued together, down where a header and footer were being counted as your content. On the first site measured, half of every page's text was furniture. Pages that only looked substantial because of a large footer will now score lower on the 300-word check, and that is the correction, not a penalty. The 'crawlers see an empty page' test deliberately still reads the whole body, so no site is newly accused of being invisible. Your generated llms-full.txt is the visible beneficiary: it is now readable prose rather than one run-on line per page with your menu repeated throughout.
- rubric v1.2.0
The rubric now credits the schema types we actually recommend
Our per-sector advice told documentation sites to publish TechArticle and APIReference, while the scoring only recognised a fixed list that included neither. A site could follow our advice exactly and score no better for it. The two lists are now reconciled, and a check in the build confirms there is no gap between what we recommend and what we count.
Effect on scoresRaises scores for sites already publishing sector-appropriate markup, including ours by 1 point. No site scores lower.
Sources: Schema.org
- rubric v1.1.0scanner v1.4.0
Attribution no longer requires a named person
The trust check looked for a byline, a Person entity, or a 'by Firstname Lastname' pattern. That quietly penalised every site attributing its content to the organisation publishing it — which is normal and correct for company documentation. It now also accepts an author declared in JSON-LD, or visible 'maintained by' / 'reviewed by' text.
Effect on scoresRaises scores for company-published content that was previously marked down for not having a personal byline. Ours rose 2 points. No site scores lower.
Sources: JSON-LD 1.1
- scanner v1.3.0
Pages your robots.txt disallows are no longer sampled
We were fetching and scoring pages sites explicitly tell crawlers to skip — sign-in and registration forms, typically. Impolite, and it measured the wrong thing: content quality judged partly on a page no crawler will ever read.
Effect on scoresRaises scores for any site with a disallowed admin or auth area, which is most of them. Ours rose 4 points. Also reduces the number of requests we make to your server.
Sources: RFC 9309: Robots Exclusion Protocol
- scanner v1.2.0
Duplicate pages, JavaScript detection, and quote verification
Redirects could collapse two sampled URLs onto one page, counting it twice and skewing every 'N of M pages' proportion. The JavaScript-dependency check fired on pages with real text and was tightened. And the grounding gate that verifies a model's quotes was rejecting truthful evidence: models answer with several real fragments joined by an ellipsis, and typographic apostrophes did not match straight ones.
Effect on scoresFewer false criticals. The quote-verification fix in particular stopped sites being told a model could not answer questions their pages plainly answer.
- scanner v1.1.0
Page sampling favours breadth
The per-section cap only applied after half the page budget was spent, so a large blog could take most of the sample before anything describing the business was considered. Sections are also now detected past a leading locale segment, and the homepage's own links are always merged with the sitemap.
Effect on scoresScores move in both directions: the sample now reflects the site rather than its largest section.
- rubric v1.0.0scanner v1.0.0
First published rubric
100 points across machine access, discovery files, content structure, structured data and trust signals.
Effect on scoresBaseline.
The rubric version covers the scoring weights and what each check accepts. The scanner version covers anything that changes a measurement — page selection, extraction, detection. Comparisons across a scanner change are labelled as ours rather than presented as your site’s movement, on the report and in the weekly email. Full detail on how the score works.
Last reviewed . Maintained by Cite Files — corrections to [email protected].