# Cite Files > Cite Files checks whether AI assistants can reach, read and cite a website — testing edge-level crawler blocking, robots.txt, sitemaps, llms.txt and ai.txt — then generates the missing files from the site's own published text. Source: https://citefiles.com Scoring rubric v1.3.0, scanner v1.13.0. Generated from the site's own page registry, so it cannot drift from the site. --- ## Cite Files Source: https://citefiles.com/ _Enter a website address and get a Citation Readiness score out of 100, free and without an account._ Cite Files checks whether AI assistants can reach, read and cite a website, then generates the files it is missing from that site's own published text. A growing share of the questions people used to type into a search box are now asked of an assistant, and the answer is written from whatever that assistant could reach and read. If a site was not reachable, or was reachable but unreadable, it is not in the answer. This fails silently. A site looks perfect in a browser while its CDN returns a bot challenge to ClaudeBot, or its platform answers 200 OK with an HTML page for /llms.txt so every checker reports a file that does not exist. Free without an account: a Citation Readiness score out of 100, the category breakdown, and the top three problems. With a free account: every finding with its evidence, a prioritised fix list, both model tests in full, and generated llms.txt, llms-full.txt, ai.txt, robots.txt, JSON-LD, an AGENTS.md template and an MCP starting point. --- ## Free AI crawler and llms.txt tools Source: https://citefiles.com/tools _Five free single-purpose checks, no account: validate an llms.txt, generate one, test whether an AI crawler is blocked, test a robots.txt for AI agents, and build an ai.txt._ Five free tools, each running one check from the full Cite Files scan, with no account. The llms.txt validator grades an llms.txt. The llms.txt generator builds one from a site's own page titles and descriptions. The AI crawler access checker requests a homepage as a browser and as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. The robots.txt tester gives an allow or deny for each AI crawler token, including Google-Extended. The ai.txt generator builds an ai.txt from a robots.txt. A tool fetches a site as CiteFilesBot, follows redirects only within the same site and never fetches pasted text; each page states its request cap. We keep a count of how many times each tool was run each day. To enforce the hourly limit, your IP address is held in a rate-limit counter that is deleted within two days. The address you test, the text you paste and the result are not stored. --- ## llms.txt validator: check your llms.txt file Source: https://citefiles.com/tools/llms-txt-validator _Paste an llms.txt or enter a site address and get the file graded, with each problem listed. Free, no account._ The llms.txt validator grades an llms.txt file and lists each problem it finds. Paste the text, or enter a site address and it fetches that site's /llms.txt. When it fetches, it requests that one file as CiteFilesBot: one request, or at most three if your own site redirects, for example from http to https to www. A redirect to another site is reported, not followed. The file is requested directly, without reading robots.txt first, because it is published for crawlers. A pasted file is never fetched. We keep a count of how many times each tool was run each day. To enforce the hourly limit, your IP address is held in a rate-limit counter that is deleted within two days. The address you test, the text you paste and the result are not stored. Each connection is limited to 30 runs an hour. --- ## llms.txt generator: build one from your own pages Source: https://citefiles.com/tools/llms-txt-generator _Enter a site address and get an llms.txt built only from that site's own page titles and meta descriptions. Free, no account._ The llms.txt generator writes an llms.txt from a site's own page titles and meta descriptions. It uses no language model, so the same site gives the same file and nothing is invented. As CiteFilesBot it reads /robots.txt, the homepage, one sitemap and up to ten pages, honouring robots.txt before reading pages. If the sitemap is an index, it takes pages from links on the homepage instead. Redirects are followed only within the same site. The hard cap is 20 requests, redirects included; at the cap it stops and says so. We keep a count of how many times each tool was run each day. To enforce the hourly limit, your IP address is held in a rate-limit counter that is deleted within two days. The address you test, the text you paste and the result are not stored. Each connection is limited to 6 runs an hour. --- ## Is GPTBot blocked on my site? AI crawler access checker Source: https://citefiles.com/tools/ai-crawler-checker _Enter a site address and see whether GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot are served or blocked, beside what robots.txt says. Free, no account._ The AI crawler access checker answers whether GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot can fetch a site. It requests the homepage as a desktop browser and as each of the four agents, plus /robots.txt, and compares what came back. The agent requests skip robots.txt, because the test is how the server answers. OpenAI and Perplexity publish the strings for GPTBot, OAI-SearchBot and PerplexityBot; Anthropic documents only the ClaudeBot token, so that string is the form seen in server logs. Redirects stay within your own site; at most 18 requests in total. We keep a count of how many times each tool was run each day. To enforce the hourly limit, your IP address is held in a rate-limit counter that is deleted within two days. The address you test, the text you paste and the result are not stored. Each connection is limited to 10 runs an hour. --- ## robots.txt tester for AI crawlers Source: https://citefiles.com/tools/robots-txt-ai-tester _Paste a robots.txt or enter a site address and see, agent by agent, whether each AI crawler is allowed or disallowed. Free, no account._ The robots.txt tester shows, for each AI crawler user-agent, whether a robots.txt allows or disallows it, and which rule decided, read the way RFC 9309 describes. Paste the text, or enter a site address and it fetches that site's /robots.txt. When it fetches, it requests that one file as CiteFilesBot: one request, or at most three if your own site redirects. A redirect to another site is reported, not followed. The file is requested directly, without reading robots.txt first. A pasted file is never fetched. We keep a count of how many times each tool was run each day. To enforce the hourly limit, your IP address is held in a rate-limit counter that is deleted within two days. The address you test, the text you paste and the result are not stored. Each connection is limited to 30 runs an hour. --- ## ai.txt generator Source: https://citefiles.com/tools/ai-txt-generator _Paste a robots.txt or enter a site address and get an ai.txt that states the same position on AI training. Free, no account._ The ai.txt generator writes an ai.txt from the position a site has already stated in its robots.txt, rather than choosing a licensing stance for the owner. Paste a robots.txt, or enter a site address and it fetches that site's /robots.txt. ai.txt is a convention, not a ratified standard, and records intent without enforcing anything. When it fetches, it requests that one file as CiteFilesBot: one request, or at most three if your own site redirects. A redirect to another site is reported, not followed. A pasted file is never fetched. We keep a count of how many times each tool was run each day. To enforce the hourly limit, your IP address is held in a rate-limit counter that is deleted within two days. The address you test, the text you paste and the result are not stored. Each connection is limited to 30 runs an hour. --- ## How the AI citation readiness score works Source: https://citefiles.com/methodology _The full rubric — every check, its weight, and how each one is measured — plus the Agent Commerce Readiness tiers, a separate unscored audit of whether an agent can transact with the site._ The Citation Readiness rubric is 100 points across five categories: Machine access (30), Discovery files (25), Content structure (20), Structured data (15) and Trust signals (10). Access dominates because nothing else matters if the crawler never receives the page. Grades: A is 85 and above, B is 70, C is 55, D is 40, F below that. The crawler access test requests the homepage as a current desktop browser, then again as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, and compares the answers structurally — status class, response size relative to the browser response, and whether the site's own title survived — rather than by vendor wording alone; known block-page wording only corroborates a structural difference. The strings are the ones OpenAI and Perplexity publish for GPTBot, OAI-SearchBot and PerplexityBot; Anthropic documents the ClaudeBot token but not a full string, so ClaudeBot is sent in the form seen in server logs. Google-Extended is not requested: Google documents it as a robots.txt token with no user agent of its own. robots.txt is read the way RFC 9309 says a crawler reads it: the rules from every group that names the crawler's token are combined (a token matches whole, ignoring case); if no group names it, the rules of the asterisk groups are combined instead; the longest matching rule decides, and Allow beats Disallow on equal length. Who your robots.txt disallows decides how a file we did not request is scored. We request none of the paths it disallows for CiteFilesBot, with one exception: the crawler-access test asks for your homepage as a browser and as each AI crawler, and follows your homepage's own redirect within your site wherever it leads, because the page a visitor lands on is the thing being measured. If it disallows a path such as /llms.txt for CiteFilesBot and the asterisk rules (the rules for every crawler without a group of its own) disallow it too, AI assistants cannot read the file either, so it scores as missing and the report says why; we do not request it. A group for CiteFilesBot that allows the path, or says nothing about it, applies to us instead of the asterisk rules, so then we read and score the file normally. If it disallows the path only for CiteFilesBot, we do not request it and cannot judge it: the check is left out, the remaining checks in that category are scaled to the category's full weight, the report marks the total a "partial score" and lists what was not measured, and the scan is left out of peer benchmarks and corpus medians. If every check in a category is left out, the category is shown as not scored and the total is scaled over the remaining categories. This can make a partial score higher than a full one would have been, which is why it is labelled and never compared. File detection guards against the phantom-file problem. Many sites answer 200 OK with their HTML shell for any unrecognised path, so before we judge any file we request a deliberately impossible path and compare every subsequent 200 against whatever that returned. The answerability test gives a model only the text a crawler receives and asks seven questions a customer would ask. An answer counts only if the model quotes the source and that quote is verified server-side to occur in what was sent. The bait test asks what a model says when the pages fall short: one model describes the organisation with no grounding instruction, a second must quote the site for each statement, and the quote is checked. Two version numbers are published: the rubric version covers the scoring weights, and the scanner version covers anything that changes a measurement. Comparisons across a scanner version change are labelled as ours rather than presented as the site's own movement. No score is ever set by a language model. Agent Commerce Readiness is a second, separate audit on the same scan, run only when an owner asks for it: eight read-only checks — agent access to the commerce page, price legibility, rate-limit and error shape, robots.txt vs the edge, machine credentials (OAuth), uCP manifest, openAPI document and mCP endpoint — bundled by published predicate into one of 6 tiers (T0 Not agent-readable, T1 Agent-readable, T2 Agent-operable, T3 Agent-integratable, T4 Agent-transactable, T5 Agent-payable), or left ungraded when the browser baseline itself could not be read. It carries its own version number, never moves the Citation Readiness score, makes never more than 40 requests, and produces a tier rather than a number, because a number we could not defend is worse than none. --- ## What has changed in the scoring Source: https://citefiles.com/methodology/changelog _Every change to the rubric and scanner, dated, with an honest note on which way it moved scores._ A versioned score is only worth anything if you can find out what the versions mean; otherwise "your score went up" is a claim nobody can check. Several changes in the record were found by running our own tools on our own site, and each happened to raise our own score. Each is defensible on its merits and the reasoning is published, but "trust us" is not something anyone should accept from the party doing the scoring, so the record exists instead. Every entry states which way it moved scores. The rubric version covers scoring weights and what each check accepts. The scanner version covers anything that changes a measurement — page selection, extraction, detection rules. Comparisons across a scanner change are labelled as ours rather than presented as the site's own movement. --- ## About Cite Files Source: https://citefiles.com/about _Why the project exists, and the three things it will not do._ Cite Files is a small independent project, not a venture-backed platform. It runs on its own hardware, including the language models, so nothing scanned is sent to a third-party model provider. Three commitments. We do not invent facts about a business: every line of a generated file traces to text the site already publishes, and anything we cannot source is emitted as a visible TODO rather than a confident guess. We do not bypass anything: we identify ourselves honestly, request only public pages, do not solve challenges, do not rotate addresses and do not touch anything behind a login. We do not tell anyone to open the gates: blocking AI crawlers is a legitimate position, and we report what current settings actually do rather than what they should be. From 3 November 2026, CiteFilesBot will also measure a site nobody asked us about: our published sample. For those, as for a scan somebody requests, robots.txt decides before anything is measured: a sample visit will begin with robots.txt, and a requested scan asks for the homepage once, only to find the site's address, and then reads robots.txt. Apart from that one homepage request, a disallow of the homepage for CiteFilesBot means no other request is made, and a challenge is recorded, never solved. --- ## CiteFilesBot: what it fetches and how to block it Source: https://citefiles.com/bot _What our crawler fetches, its user-agent string, and how to block it._ CiteFilesBot visits a site for one of two reasons. Somebody asked: a person, usually the site's owner, ran a scan, used one of the free tools, asked an AI agent to use our MCP server, or set up a weekly watch. Those visits identify as: CiteFilesBot/1.0 (+https://citefiles.com/bot; audit requested by a visitor) Our published sample: from 3 November 2026, so that the open record describes more than the sites people chose to scan, CiteFilesBot will also measure a fixed, dated list of well-known sites on a schedule, about once a month each. The sample is a fixed list of about a thousand well-known sites, published on the record when the first edition starts, minus the exclusions published with it. Those visits will identify as: CiteFilesBot/1.0 (+https://citefiles.com/bot; published sample) When somebody asks, a scan requests your homepage once to find its address, then reads robots.txt before anything else, then the sitemap, about a dozen conventional file paths, up to twenty-five pages, and one deliberately impossible address so a real file can be told apart from a page your server returns for everything, requesting none of the paths robots.txt disallows for CiteFilesBot, with one exception: the crawler-access test asks for your homepage as a browser and as each AI crawler, and follows your homepage's own redirect within your site wherever it leads, because the page a visitor lands on is the thing being measured. If robots.txt disallows CiteFilesBot from the homepage, the scan stops there and reports that it could not read the site. If robots.txt answers with a server error or does not answer, the scan treats that as a disallow, as RFC 9309 asks, and stops; if it asks us to slow down (429), the scan stops too. Two optional extras, which the scan's owner has to ask for, also identify as CiteFilesBot, and each reads your robots.txt first: the technical checks use the copy the scan read if it is under a day old and read it afresh otherwise, and the "Check your published files" button reads it afresh every time. If it cannot be read, both treat that as a disallow. The technical checks look at up to 40 of the links on your own site that your homepage points to, skipping any your robots.txt disallows for CiteFilesBot. Each link gets a HEAD request; if your server answers 405 or 501, or does not answer, we send one GET before calling the link broken. The button reads robots.txt and then requests five fixed paths once each with a GET, /llms.txt, /llms-full.txt, /ai.txt, /robots.txt and /AGENTS.md, to see whether the files we generated are live and correctly served. It skips any of those paths your robots.txt disallows for CiteFilesBot, except /robots.txt itself, which it always requests. Both follow up to five redirects, other than into a path your robots.txt disallows for CiteFilesBot. The full report and the technical checks also open a few of the pages the scan already read in a real browser, at most four for the report and three for the technical checks, to see what JavaScript adds and to run Lighthouse and an accessibility check. A browser loads what those pages load, so those visits also fetch their scripts, styles and images. The technical checks skip a page your robots.txt now disallows for CiteFilesBot. One exception, stated plainly: the crawler-access test deliberately sends the user-agent strings OpenAI and Perplexity publish for GPTBot, OAI-SearchBot and PerplexityBot, and the string ClaudeBot is seen sending (Anthropic documents the token, not the full string), because that is the only way to observe how an origin treats those agents. Those four requests fetch the public homepage and nothing else, and we make no attempt to defeat any challenge we receive. The single-purpose tools under /tools fetch less than a scan, and follow a redirect only within the same site. The llms.txt validator, the robots.txt tester and the ai.txt generator each request one file, /llms.txt or /robots.txt: one request, or at most three if your own site redirects. The validator and the tester request that file directly, without consulting robots.txt first, because it is a file published for crawlers. The AI crawler access checker requests the homepage as a browser and as each of the four agents above, plus /robots.txt, at most 18 requests in total. The llms.txt generator reads /robots.txt, the homepage, one sitemap and up to ten pages, and honours robots.txt before reading pages; if the sitemap is an index it takes pages from links on the homepage instead. Its hard cap is 20 requests, redirects included. A pasted input is never fetched. A scheduled visit to a site in the published sample will begin with robots.txt. Before a scheduled visit we will request your robots.txt as CiteFilesBot, and if it disallows CiteFilesBot from your homepage, in a group for CiteFilesBot or in the wildcard group when there is none, we will make no further request and will not measure the site. If robots.txt answers with a server error or does not answer, we will treat that as a disallow, as RFC 9309 asks; if it asks us to slow down (429), we will stop; either way we will try another day. We will read it fresh every time. After that: your homepage once as CiteFilesBot, to follow your own redirect (and robots.txt again if the redirect lands on a different host name); your homepage once as a desktop browser and once as each of the four crawlers named above; and /llms.txt plus one deliberately impossible address, so a real file can be told apart from a page your server returns for everything. At most 24 requests counting redirects, one after another. If robots.txt sets a Crawl-delay for CiteFilesBot we will wait that long between requests, up to ten seconds. Within 72 hours a second server of ours, on a different network address, will repeat the robots.txt check, the browser request and the four crawler requests; we will publish a result about a site only when both servers got the same answer. To block us, add to robots.txt: User-agent: CiteFilesBot then Disallow: / — we honour it. A scan somebody asks for requests your homepage once to find its address, then robots.txt. If robots.txt disallows CiteFilesBot from your homepage, the scan stops there and reports that it could not read the site; if it disallows only particular paths, the scan does not request those paths (the crawler-access test still follows your homepage's own redirect). A site in our published sample will have its robots.txt re-checked at least every 7 days; when we find the disallow we will stop measuring it and take it out of the record, its page and the downloadable data within 7 days. What we keep from the published sample: for a site in it we will keep the measurement only: response codes and sizes, the homepage title, the first 4,000 characters of robots.txt, which crawlers were served or refused, and what the edge identified itself as. No page text. We will keep the latest three measurements of each site, plus any measurement a dated report cites. To be taken out, disallow CiteFilesBot as above or write to abuse@citefiles.com from an address at your domain: we will remove it within two working days and will not ask why. A dated report will keep its totals, say "withheld at the site's request", and drop the row from its download. --- ## Which websites block AI crawlers? An open record Source: https://citefiles.com/index/crawler-access _An open record of how real websites treat GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot._ Every site scanned is asked for its homepage twice: once as a current desktop browser, once as each crawler. This page publishes what came back, using the most recent scan of each site. Two populations, never mixed. From 3 November 2026 the published sample will be a fixed list of about a thousand well-known sites, published on the record when the first edition starts, minus published exclusions, measured on a schedule after robots.txt allows it. The other is sites somebody chose to scan, which skews towards sites whose owners suspected a problem, so a refusal rate there is a fact about those sites rather than a measurement of the internet. A refusal is not necessarily a mistake: blocking AI crawlers is a legitimate position, and what this records is what a server actually does, which is frequently not what its owner believes it does. Every crawler request is sent from our servers, not the vendors' networks, so a site that checks where a request comes from may refuse us and still serve the real crawler. Each result for a site somebody chose to scan is one request per crawler from one of our servers. Each result for a sample site will be repeated once from a second server of ours before it is published. Neither kind of result can see rules that depend on volume, geography or the vendor's own network. Only sites in the published sample will be named, and only when two of our servers on different network addresses got the same answer within 72 hours. Sites people chose to scan are counted, never named. Removal: disallow CiteFilesBot in robots.txt (which will be re-checked at least every 7 days for sites in the sample), or write to abuse@citefiles.com from an address at that domain; we remove it within two working days and do not ask why. --- ## Which sites can an AI agent actually transact with Source: https://citefiles.com/index/agent-commerce _An open record of Agent Commerce Readiness tiers, T0 to T5 — whether an AI agent can read prices, call an API and obtain its own credentials — across sites whose owners chose to run the check._ Agent Commerce Readiness is a ladder of 6 tiers, T0 to T5, each one a bundle of reproducible facts rather than a score. T0, Not agent-readable, means agents are refused the commerce pages, or no price is readable without running the page. T1, Agent-readable, means an agent gets the commerce page, and the prices are text the server sent. T2, Agent-operable, means an unauthenticated API request is answered with something a machine can act on (rate-limit headers, a machine-readable error, or a structured JSON answer), and what robots.txt says about agents agrees with what the edge does. T3, Agent-integratable, means a real machine interface exists — a parseable OpenAPI document, a live MCP endpoint, or a valid UCP manifest. Any one. T4, Agent-transactable, means an agent can obtain its own credentials — client_credentials declared explicitly, or dynamic client registration. T5, Agent-payable, means the site answers with a machine-payable challenge — x402 (v1 or v2) or L402 — so an agent that holds credentials could settle without a human. We observe the challenge; we never settle it. Ungraded is not a tier. A site whose browser baseline could not be read produced no evidence at all, and is excluded from the distribution rather than counted as T0 — T0 is itself a positive claim, that we looked and found agents refused or prices illegible, never a default for a site we could not reach. Every audit here ran because a site's owner asked for it, which is its own selection effect: this is not a random sample of the web, and the page says so wherever a figure appears. Site names are published only when that has been deliberately enabled, and a tier is a stronger claim than a crawler-access result, so no host is named here until its tier has been reproduced by hand from a different network than the one that measured it. Any named site can be removed on request from an address at that domain, without being asked why. --- ## Agent Commerce Readiness: can an AI agent buy from you? Source: https://citefiles.com/agent-commerce-readiness _What Agent Commerce Readiness measures — eight read-only checks bundled into a tier, T0 to T5 — how to run it on a site you own, and how to check every predicate yourself with curl._ Agent Commerce Readiness is a second, unscored audit on the same scan as the Citation Readiness score: whether an AI agent could transact with a site, not just read it. Eight read-only checks — agent access to the commerce page, price legibility, rate-limit and error shape, robots.txt vs the edge, machine credentials (OAuth), uCP manifest, openAPI document and mCP endpoint — are bundled by published predicate into one of 6 tiers. T0, Not agent-readable, means agents are refused the commerce pages, or no price is readable without running the page. T1, Agent-readable, means an agent gets the commerce page, and the prices are text the server sent. T2, Agent-operable, means an unauthenticated API request is answered with something a machine can act on (rate-limit headers, a machine-readable error, or a structured JSON answer), and what robots.txt says about agents agrees with what the edge does. T3, Agent-integratable, means a real machine interface exists — a parseable OpenAPI document, a live MCP endpoint, or a valid UCP manifest. Any one. T4, Agent-transactable, means an agent can obtain its own credentials — client_credentials declared explicitly, or dynamic client registration. T5, Agent-payable, means the site answers with a machine-payable challenge — x402 (v1 or v2) or L402 — so an agent that holds credentials could settle without a human. We observe the challenge; we never settle it. Ungraded is not a tier: it means the browser baseline itself could not be read, so nothing could be compared. It never moves the Citation Readiness score, never submits a form or attempts a payment, and makes never more than 40 requests, honouring robots.txt for every path it touches. Only the signed-in owner of a scan can run it, from that scan's report page, and only by choosing to. Q: What is Agent Commerce Readiness? A: A second audit on the same scan. The Citation Readiness score asks whether an AI assistant can reach, read and cite a site; this asks whether an AI agent could transact with it. Eight read-only checks are bundled into a tier, T0 up to T5, by published predicates. There is no score, on purpose. Q: Does it change my Citation Readiness score? A: No. It has its own rubric version, its own table and its own predicates, and nothing in it feeds the score. Running it moves no number anywhere. Q: Who can run it, and does it run on its own? A: Only the signed-in owner of a scan, from the report page, and only when they ask. It never runs automatically on a site. Q: What does it do to my server? A: Never more than 40 read-only requests, usually well under that. It honours your robots.txt for every path, sends exactly one POST (an MCP server/discover, and only after a GET has answered 405), and never submits a form or a payment. Q: What do the tiers mean? A: T0 Not agent-readable, T1 Agent-readable, T2 Agent-operable, T3 Agent-integratable, T4 Agent-transactable, T5 Agent-payable — each rung a bundle of reproducible facts decided by a published predicate, checkable with curl; the full predicate for each is on the methodology page. Ungraded is not a tier: it means the browser baseline itself was refused or a first-rung cell could not be read, so nothing could be compared. Q: Is Ungraded a bad grade? A: No. Ungraded is not a tier. It means the browser baseline itself was refused or a first-rung cell could not be read, so nothing could be compared. A site we could not measure has produced no evidence, and T0 is a claim we only make when we looked and found agents refused or prices illegible. Q: What does citefiles.com grade on its own rubric? A: T0, Not agent-readable. Our pricing page is /pro, and it carries no price yet — it describes a product that is not for sale — so the first rung fails honestly: the page was found, read as a browser and as each agent, and no price appears in it. Our MCP endpoint answers 405 to a GET and describes itself to a server/discover request, so T3's interface predicate would be met, but the ladder is climbed rung by rung and T1 is not. Until this build the same site read Ungraded, because the scanner only recognised a pricing page by the words pricing, plans or price in its path; the changelog records the change. Q: What is DIVERGENT? A: A site can serve an agent a real commerce page whose price or title differs from what a browser sees at the same URL. That site is still served, so it can still climb the ladder; the flag is raised beside the tier with both sets of prices shown, and the reader judges. It never moves the tier. Q: Does it test whether an agent can actually pay? A: No — we never settle a challenge. T5 means the site ANSWERS with a machine-payable challenge (x402 or L402) that a credentialed agent could settle; that is an offer we can see with one request, not a transaction. Cloudflare's pay-per-crawl and a bare 402 with no machine-readable challenge stay recorded as watch items, never graded, and completing a trade leaves nothing a GET can see. Q: What are UCP, MCP, OpenAPI and x402, in one line each? A: UCP is a public manifest at /.well-known/ucp describing a merchant's agent-commerce services. MCP is the Model Context Protocol, an endpoint that answers 405 to GET and describes itself to one server/discover request. OpenAPI is a JSON description of an HTTP API with an openapi or swagger key. x402 (v1 or v2) is an HTTP 402 carrying a machine-readable challenge in a header that a credentialed agent could settle without a human; L402 is the same idea over a Lightning macaroon. We record the challenge shape and grade it at T5; we never settle one. Q: How do I check a predicate myself? A: Every predicate is a curl. Request your pricing page with a browser user-agent and again with an agent's published user-agent string and compare status, size and title. Fetch /.well-known/oauth-authorization-server and read grant_types_supported: client_credentials must appear explicitly, because the RFC 8414 default excludes it. GET your MCP path and expect 405; anything else is not an MCP endpoint by the spec. Q: Why is the T2 rung about robots.txt at all? A: Because robots.txt is what a site says about agents and the edge is what it does, and the two are allowed to disagree. A robots.txt that allows agents while the edge challenges them is a contradiction, and only the enforced one is what an agent meets. Coherent-closed is honest and fails T2 honestly. Q: How do I remove my site, or stop the check running? A: Write to abuse@citefiles.com from an address at that domain and we remove it without asking why. To stop the check reading your site, disallow CiteFilesBot in robots.txt; the audit honours it for every path. Q: Why is the public record empty? A: It publishes nothing below eight graded sites, and naming a host on it needs two things at once: its tier reproduced by hand from a different network than the one that measured it, recorded against that exact tier and rubric version, and this page's publishing switch turned on. A later audit that moves the tier, or a rubric-version bump, silently un-names the host. Every audit in it ran because an owner asked, so it is a fact about that sample and not about the web. --- ## AI citation readiness FAQ: llms.txt, ai.txt, AI crawlers Source: https://citefiles.com/faq _Plain answers about the Citation Readiness score and the Agent Commerce Readiness tier: what is measured, what is not, and how to check it yourself._ Citation Readiness questions. Q: What does the Citation Readiness score measure? A: One hundred points across five categories: machine access (30), discovery files (25), content structure (20), structured data (15) and trust signals (10). Every point is earned on something we measured — a request we sent, a file we found, a heading we counted — never on a model's opinion. Grades run from A at 85 and above down to F below 40. Q: Why does machine access carry the most points? A: Fourteen of access's thirty points come from whether AI crawlers are actually served at the edge, because nothing else in the rubric matters if the crawler never receives the page. A flawless llms.txt behind a WAF that answers 403 to ClaudeBot is worth nothing at all, so the precondition is weighted heaviest, not the polish on top of it. Q: What does 'turned away at the edge' mean, and how do you detect it? A: We request your homepage once as a current desktop browser, then again as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, and compare each crawler's answer to the browser's. We send the user-agent strings OpenAI and Perplexity publish for GPTBot, OAI-SearchBot and PerplexityBot, and the string ClaudeBot is seen sending, because Anthropic documents the token but not a full string. Google-Extended is not requested: Google documents it as a robots.txt token with no user agent of its own, so we read its rule from robots.txt instead. Blocked means an error status where the browser got a page. Challenged means an interstitial — a bot-check page crawlers do not solve — decided structurally: status class, response size relative to the browser response, and whether your own title tag survived, corroborated by known challenge-page phrases rather than keyword-matched alone. Q: What does the free summary show, and what does a free account add? A: Without an account: a Citation Readiness score out of 100, the category breakdown, and the top three problems. With a free account: every finding with its evidence, a prioritised fix list, both model tests in full, and generated llms.txt, llms-full.txt, ai.txt, robots.txt, JSON-LD, an AGENTS.md template and an MCP starting point. Q: How are llms.txt and llms-full.txt graded, and what do you generate? A: llms.txt is worth ten points: four just for being present and genuine — not a 200 OK HTML shell mistaken for a file — and up to six more scaled by a quality score out of 100 and its link count. llms-full.txt is worth five: three for existing, five if it also carries 300 or more words and at least one heading. With a free account we generate both from your site's own published text, alongside ai.txt, robots.txt, JSON-LD, an AGENTS.md template and an MCP starting point. Q: What are the answerability test and the bait test, and how are quotes verified? A: The answerability test hands a model only the text a crawler receives from your pages and asks seven fixed questions a customer would ask — what you do, who it's for, where you operate, how to reach you, what it costs, who runs it, how current it is. An answer only counts if the model returns a verbatim quote, and we check server-side that the quote actually occurs in the text we sent; if not, the model was filling a gap, and the question is marked unanswerable. The bait test asks the opposite: with no grounding instruction, one model describes your organisation the way it would to someone who asked, a second model must find a verbatim quote supporting each statement, and we verify that quote the same way. Q: What does CiteFilesBot fetch, and how do I block it? A: Visits somebody asked for identify as CiteFilesBot/1.0 (+https://citefiles.com/bot; audit requested by a visitor): a scan asks for your homepage once to find the site's address, then reads robots.txt and stops if CiteFilesBot is disallowed from the homepage, or robots.txt errors or does not answer (RFC 9309 treats that as a disallow), or answers 429. It then fetches the sitemap, about a dozen conventional files and up to twenty-five pages, skipping any path robots.txt disallows. From 3 November 2026 a published sample of about a thousand sites will be measured about monthly, robots.txt first, at most 24 requests, as CiteFilesBot/1.0 (+https://citefiles.com/bot; published sample). The crawler-access test sends the user agents of GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot to your homepage from our servers. To block us, add 'User-agent: CiteFilesBot' and 'Disallow: /' to robots.txt. Q: What does running a scan do to my server? A: About fifty read-only GET requests, a few at a time and no more than twenty-five of them pages, usually well under a minute of traffic. We identify ourselves honestly, request only public pages, do not solve challenges, do not rotate addresses, and do not touch anything behind a login. No forms are submitted and nothing is written to your site — the whole audit is arithmetic on things we fetched, never an action taken against it. Q: How do weekly watches work, and when do you email? A: You can ask us to re-check a site once a week. We email you only when something material moved: the score shifted by two or more, a crawler started being turned away, a file appeared or vanished, or a new serious problem exists — never as a routine heartbeat, because a monitoring message that usually says nothing gets filtered, and then the one that matters gets filtered with it. Every scan records which scanner version produced it, so a change caused by us improving how we measure is labelled as ours rather than shown as your site's movement. A watch that keeps failing backs off and stops after five consecutive failures. Q: How long do you keep my data, and how do I have it deleted? A: Scans not attached to an account are deleted after 90 days, enforced by a scheduled job. Scans in an account stay until you delete them or close the account. You can delete your own account immediately from your account page — that removes your scans, reports, watches, batches, badges and feedback and cannot be undone — or write to hello@citefiles.com and we act within 30 days. We do not sell or share what we hold, and nothing you scan is sent to a third-party model provider. Q: How do I remove my site from the public crawler-access record? A: Write to abuse@citefiles.com from an address at the domain in question and we remove it within two working days, without asking why. For a site in our published sample you can instead disallow CiteFilesBot in robots.txt: from 3 November 2026 that file will be re-checked at least every 7 days, and when we find the disallow we will stop measuring the site and take it out. A site will only ever be named on the record if it is in the published sample and two of our servers, on different network addresses, got the same answer within 72 hours. Q: Why did CiteFilesBot visit my site when nobody asked? A: Before 3 November 2026 every visit was requested by somebody, because anyone can ask for a scan of a public site. From that date, a visit whose user-agent string says 'published sample' will be part of our scheduled sample: about a thousand well-known sites, a fixed list published on the record when the first edition starts, each to be measured about once a month, so the open record of AI crawler access is not only the sites people chose to scan. Sample visits will read robots.txt first, and a disallow for CiteFilesBot, or a robots.txt that errors or does not answer, will end the visit. To opt out, disallow CiteFilesBot in robots.txt or write to abuse@citefiles.com from an address at your domain. Q: Why are your own scores and tiers published on your own site? A: Because a versioned score is only meaningful if you can check what changed and why, and 'trust us' is not something you should accept from the party doing the scoring. Several rubric fixes were found by running our own tools on our own site, and each happened to raise our own score — one point here, two there, four elsewhere — so instead of asserting each was defensible we publish the changelog and let you read the reasoning yourself. Agent Commerce Readiness questions. Q: What is Agent Commerce Readiness? A: A second audit on the same scan. The Citation Readiness score asks whether an AI assistant can reach, read and cite a site; this asks whether an AI agent could transact with it. Eight read-only checks are bundled into a tier, T0 up to T5, by published predicates. There is no score, on purpose. Q: Does it change my Citation Readiness score? A: No. It has its own rubric version, its own table and its own predicates, and nothing in it feeds the score. Running it moves no number anywhere. Q: Who can run it, and does it run on its own? A: Only the signed-in owner of a scan, from the report page, and only when they ask. It never runs automatically on a site. Q: What does it do to my server? A: Never more than 40 read-only requests, usually well under that. It honours your robots.txt for every path, sends exactly one POST (an MCP server/discover, and only after a GET has answered 405), and never submits a form or a payment. Q: What do the tiers mean? A: T0 Not agent-readable, T1 Agent-readable, T2 Agent-operable, T3 Agent-integratable, T4 Agent-transactable, T5 Agent-payable — each rung a bundle of reproducible facts decided by a published predicate, checkable with curl; the full predicate for each is on the methodology page. Ungraded is not a tier: it means the browser baseline itself was refused or a first-rung cell could not be read, so nothing could be compared. Q: Is Ungraded a bad grade? A: No. Ungraded is not a tier. It means the browser baseline itself was refused or a first-rung cell could not be read, so nothing could be compared. A site we could not measure has produced no evidence, and T0 is a claim we only make when we looked and found agents refused or prices illegible. Q: What does citefiles.com grade on its own rubric? A: T0, Not agent-readable. Our pricing page is /pro, and it carries no price yet — it describes a product that is not for sale — so the first rung fails honestly: the page was found, read as a browser and as each agent, and no price appears in it. Our MCP endpoint answers 405 to a GET and describes itself to a server/discover request, so T3's interface predicate would be met, but the ladder is climbed rung by rung and T1 is not. Until this build the same site read Ungraded, because the scanner only recognised a pricing page by the words pricing, plans or price in its path; the changelog records the change. Q: What is DIVERGENT? A: A site can serve an agent a real commerce page whose price or title differs from what a browser sees at the same URL. That site is still served, so it can still climb the ladder; the flag is raised beside the tier with both sets of prices shown, and the reader judges. It never moves the tier. Q: Does it test whether an agent can actually pay? A: No — we never settle a challenge. T5 means the site ANSWERS with a machine-payable challenge (x402 or L402) that a credentialed agent could settle; that is an offer we can see with one request, not a transaction. Cloudflare's pay-per-crawl and a bare 402 with no machine-readable challenge stay recorded as watch items, never graded, and completing a trade leaves nothing a GET can see. Q: What are UCP, MCP, OpenAPI and x402, in one line each? A: UCP is a public manifest at /.well-known/ucp describing a merchant's agent-commerce services. MCP is the Model Context Protocol, an endpoint that answers 405 to GET and describes itself to one server/discover request. OpenAPI is a JSON description of an HTTP API with an openapi or swagger key. x402 (v1 or v2) is an HTTP 402 carrying a machine-readable challenge in a header that a credentialed agent could settle without a human; L402 is the same idea over a Lightning macaroon. We record the challenge shape and grade it at T5; we never settle one. Q: How do I check a predicate myself? A: Every predicate is a curl. Request your pricing page with a browser user-agent and again with an agent's published user-agent string and compare status, size and title. Fetch /.well-known/oauth-authorization-server and read grant_types_supported: client_credentials must appear explicitly, because the RFC 8414 default excludes it. GET your MCP path and expect 405; anything else is not an MCP endpoint by the spec. Q: Why is the T2 rung about robots.txt at all? A: Because robots.txt is what a site says about agents and the edge is what it does, and the two are allowed to disagree. A robots.txt that allows agents while the edge challenges them is a contradiction, and only the enforced one is what an agent meets. Coherent-closed is honest and fails T2 honestly. Q: How do I remove my site, or stop the check running? A: Write to abuse@citefiles.com from an address at that domain and we remove it without asking why. To stop the check reading your site, disallow CiteFilesBot in robots.txt; the audit honours it for every path. Q: Why is the public record empty? A: It publishes nothing below eight graded sites, and naming a host on it needs two things at once: its tier reproduced by hand from a different network than the one that measured it, recorded against that exact tier and rubric version, and this page's publishing switch turned on. A later audit that moves the tier, or a rubric-version bump, silently un-names the host. Every audit in it ran because an owner asked, so it is a fact about that sample and not about the web. --- ## Contact Source: https://citefiles.com/contact _How to reach us: general questions and corrections, security and removal, account problems._ A person reads every address. There is no ticket system and no chatbot. hello@citefiles.com — general questions and corrections. If a check misjudged a site, this is the address that fixes it; the rubric is versioned so corrections are traceable, and a wrong check is a bug. abuse@citefiles.com — security reports, requests to remove a scan of your own site, and requests to take your site out of our published sample (which starts on 3 November 2026). Write from an address at the domain in question. support@citefiles.com — account problems: sign-in trouble, a report that will not generate, a file that came out wrong. Security reports and removal requests are answered within two working days; everything else within a week. --- ## MCP server and GitHub Action for AI readiness checks Source: https://citefiles.com/integrations _Connect the audit to a coding agent over MCP, or to CI so a change that makes a site less citable fails the build._ An audit you have to remember to run is an audit that gets run once. Two integrations connect it to where the work happens, both free and neither needing an account. MCP: Cite Files speaks the Model Context Protocol at https://citefiles.com/mcp. A coding agent can scan a site, read the findings, change the repository and scan again to check its own work. Four tools: citefiles_scan for the score and findings, citefiles_fix_prompt for a task list with the rules that stop an agent inventing facts about the business, citefiles_check_files for which discovery files are genuinely served, and citefiles_crawler_access for how the site answers GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. The MCP tools return measurements and findings only. The written report and the generated files need a free account on the website, and the endpoint does not hand them out. No tool takes a username or password: a tool call is the wrong place for a secret, because the value ends up in an agent transcript. CI: a composite GitHub Action checks the live site on a schedule and fails the build below a score you set, or on critical findings. Sites drift — a framework upgrade drops a sitemap, a CDN rule starts refusing GPTBot, a page becomes client-rendered and its text vanishes from the HTML. An unreadable site is reported as unreadable rather than as a score of zero, and an outage at our end warns rather than failing the build. --- ## Answer tracking — coming soon Source: https://citefiles.com/pro _What a paid tier would add, why it costs money, and what stays free. Not available yet._ Everything on the site today is free and stays free: whether an assistant can reach and read a site, and whether a search engine returns it for the questions its customers ask. That runs on our own hardware at effectively no cost per report. The step after that is reading the AI answer itself. Google's AI Overview names its sources, and recording whether a site was among them — and which sites were instead, and how that moves week to week — costs a fraction of a penny per question through a data provider, and several times more to ask the named assistants directly. That is why it will be a paid option rather than something added quietly to the free tier. Two things it will not claim: an API answer from a model is not the consumer product, so any "your ChatGPT citation rate" figure is a proxy and will be labelled as one; and every measurement is a sample of one question at one moment, so a trend is strong evidence where a single check is weak. The strongest evidence of a real citation is free and already available: a request from ChatGPT-User in your own access logs is a person's assistant fetching your page because they asked it something. --- ## Report a problem or suggest something Source: https://citefiles.com/feedback _Tell us if a result is wrong about your site, if something is broken, or what to build next._ This tool makes claims about other people's websites, and a confident wrong claim is the worst thing it can do. If a result is wrong about your site we want to hear it, and no account or email address is required to say so. Reports that a result is wrong are re-run by hand before anything changes. Where the correction affects scoring it is published on the scoring changelog with its effect stated, including whose scores go up and whose go down. Two other things are useful here: something broken — a page that will not load, a file that downloads empty, a scan that hangs — and suggestions for what would make the report more useful. --- ## Privacy Source: https://citefiles.com/privacy _What is stored, for how long, and how to have it deleted._ We store the sites you scan and, with an account, your email address. Nothing is sold or shared, there are no advertising trackers, and no third-party embeds. Anonymous scans are deleted after 90 days, enforced by a scheduled job rather than only promised here. Scans in an account stay until the owner deletes them. The models that write reports run on our own hardware. Scan data is not sent to OpenAI, Anthropic, Google or any other model provider, and nothing scanned is used to train anything. The single-purpose tools under /tools keep little. We keep a count of how many times each tool was run each day. To enforce the hourly limit, your IP address is held in a rate-limit counter that is deleted within two days. The address you test, the text you paste and the result are not stored. A tool that fetches a site requests only what its own page says, as CiteFilesBot, and follows redirects only within the same site. Sites we will measure without being asked (from 3 November 2026): CiteFilesBot will measure a published sample of well-known sites about once a month, after reading their robots.txt. For those sites we will store the measurement described on /bot and no page text, keep the latest three measurements plus any a dated report cites, and remove a site on request or when its robots.txt disallows CiteFilesBot. None of this will concern visitors to those sites; we will collect nothing about the visitors to the sites we will measure. --- ## Terms Source: https://citefiles.com/terms _Terms of use in plain language, including what the report does and does not promise._ A scan is a set of measurements taken at one moment from one place. A crawler-access result reflects how an origin answered us then; it can change with CDN settings, traffic or requesting network. The report is evidence, not a warranty. Generated files are drawn from a site's own published text and are a starting point, not a finished publication. We do not promise that acting on a report will improve traffic, rankings or how often an assistant mentions a site. What we can say is whether a crawler can currently read it, which is a precondition for all of it. --- ## AI crawlers and robots.txt tokens: a sourced reference Source: https://citefiles.com/crawlers _One page per AI crawler and robots.txt token (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot, Applebot-Extended, CCBot, meta-externalagent, Bingbot), each sourced from the operator's own documentation._ This reference has one page for each of 14 AI crawlers and robots.txt tokens: GPTBot (OpenAI), OAI-SearchBot (OpenAI), ChatGPT-User (OpenAI), ClaudeBot (Anthropic), Claude-User (Anthropic), Claude-SearchBot (Anthropic), PerplexityBot (Perplexity), Perplexity-User (Perplexity), Google-Extended (Google), Applebot (Apple), Applebot-Extended (Apple), CCBot (Common Crawl), meta-externalagent (Meta), Bingbot (Bing). Each page sets out what the operator's own documentation says about the token's purpose, its user-agent string where one is published, what robots.txt does to it, what blocking it does to search where the operator says, and how to verify a request where the operator gives a method. Each statement is quoted from the operator's page and linked to it, and each page shows the date it was checked. Where an operator publishes no user-agent string, as with Google-Extended, the page says so rather than showing one. The index also gives two example robots.txt groups built from the same data, labelled as examples. --- ## GPTBot: user agent and how to block it Source: https://citefiles.com/crawlers/gptbot _OpenAI's training crawler: its published user agent, what OpenAI says about robots.txt, and how to verify requests._ GPTBot is an entry in OpenAI's crawler documentation, listed here as training. OpenAI describes it this way: “GPTBot is used to make our generative AI foundation models more useful and safe. It is used to crawl content that may be used in training our generative AI foundation models. Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models.” Its user-agent string as OpenAI publishes it is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot. On robots.txt, OpenAI says: “OpenAI uses OAI-SearchBot and GPTBot robots.txt tags to enable webmasters to manage how their sites and content work with AI.” OpenAI publishes the IP addresses for GPTBot as a JSON file. On search, OpenAI says: “Each setting is independent of the others – for example, a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training OpenAI’s generative AI foundation models.” Cite Files requests this agent when it scans a site. Checked against OpenAI's documentation on 4 October 2026. --- ## OAI-SearchBot: user agent and how to block it Source: https://citefiles.com/crawlers/oai-searchbot _OpenAI's search crawler: its published user agent, what OpenAI says about robots.txt, and how to verify requests._ OAI-SearchBot is an entry in OpenAI's crawler documentation, listed here as search. OpenAI describes it this way: “OAI-SearchBot is for search. OAI-SearchBot is used to surface websites in search results in ChatGPT’s search features.” Its user-agent string as OpenAI publishes it is Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot. On robots.txt, OpenAI says: “OpenAI uses OAI-SearchBot and GPTBot robots.txt tags to enable webmasters to manage how their sites and content work with AI.” OpenAI publishes the IP addresses for OAI-SearchBot as a JSON file. On search, OpenAI says: “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.” Cite Files requests this agent when it scans a site. Checked against OpenAI's documentation on 4 October 2026. --- ## ChatGPT-User: user agent and how to block it Source: https://citefiles.com/crawlers/chatgpt-user _OpenAI's user-triggered fetcher: its published user agent, what OpenAI says about robots.txt, and how to verify requests._ ChatGPT-User is an entry in OpenAI's crawler documentation, listed here as user-triggered. OpenAI describes it this way: “OpenAI also uses ChatGPT-User for certain user actions in ChatGPT and Custom GPTs.” Its user-agent string as OpenAI publishes it is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot. On robots.txt, OpenAI says: “Because these actions are initiated by a user, robots.txt rules may not apply.” OpenAI publishes the IP addresses for ChatGPT-User as a JSON file. On search, OpenAI says: “ChatGPT-User is not used to determine whether content may appear in Search.” Cite Files does not request this agent when it scans a site. Checked against OpenAI's documentation on 4 October 2026. --- ## ClaudeBot: robots.txt token and blocking it Source: https://citefiles.com/crawlers/claudebot _Anthropic's training crawler: its robots.txt token, what Anthropic says about robots.txt, and how to verify requests._ ClaudeBot is an entry in Anthropic's crawler documentation, listed here as training. Anthropic describes it this way: “ClaudeBot helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training.” Anthropic’s page documents the robots.txt token and publishes no full user-agent string for it, so none is shown here. On robots.txt, Anthropic says: “Anthropic’s Bots respect “do not crawl” signals by honoring industry standard directives in robots.txt.” Anthropic publishes a list of source IP addresses and says: “If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic.” On crawl rate, Anthropic says: “To limit crawling activity, we support the non-standard Crawl-delay extension to robots.txt.” Cite Files requests this agent when it scans a site. Checked against Anthropic's documentation on 4 October 2026. --- ## Claude-User: robots.txt token and blocking it Source: https://citefiles.com/crawlers/claude-user _Anthropic's user-triggered fetcher: its robots.txt token, what Anthropic says about robots.txt, and how to verify requests._ Claude-User is an entry in Anthropic's crawler documentation, listed here as user-triggered. Anthropic describes it this way: “Claude-User supports Claude AI users. When individuals ask questions to Claude, it may access websites using a Claude-User agent.” Anthropic’s page documents the robots.txt token and publishes no full user-agent string for it, so none is shown here. On robots.txt, Anthropic says: “Anthropic’s Bots respect “do not crawl” signals by honoring industry standard directives in robots.txt.” Anthropic publishes a list of source IP addresses and says: “If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic.” On search, Anthropic says: “Disabling Claude-User on your site prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search.” On crawl rate, Anthropic says: “To limit crawling activity, we support the non-standard Crawl-delay extension to robots.txt.” Cite Files does not request this agent when it scans a site. Checked against Anthropic's documentation on 4 October 2026. --- ## Claude-SearchBot: robots.txt token and blocking it Source: https://citefiles.com/crawlers/claude-searchbot _Anthropic's search crawler: its robots.txt token, what Anthropic says about robots.txt, and how to verify requests._ Claude-SearchBot is an entry in Anthropic's crawler documentation, listed here as search. Anthropic describes it this way: “Claude-SearchBot navigates the web to improve search result quality for users. It analyzes online content specifically to enhance the relevance and accuracy of search responses.” Anthropic’s page documents the robots.txt token and publishes no full user-agent string for it, so none is shown here. On robots.txt, Anthropic says: “Anthropic’s Bots respect “do not crawl” signals by honoring industry standard directives in robots.txt.” Anthropic publishes a list of source IP addresses and says: “If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic.” On search, Anthropic says: “Disabling Claude-SearchBot on your site prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results.” On crawl rate, Anthropic says: “To limit crawling activity, we support the non-standard Crawl-delay extension to robots.txt.” Cite Files does not request this agent when it scans a site. Checked against Anthropic's documentation on 4 October 2026. --- ## PerplexityBot: user agent and how to block it Source: https://citefiles.com/crawlers/perplexitybot _Perplexity's search crawler: its published user agent, what Perplexity says about robots.txt, and how to verify requests._ PerplexityBot is an entry in Perplexity's crawler documentation, listed here as search. Perplexity describes it this way: “PerplexityBot is designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models.” Its user-agent string as Perplexity publishes it is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot). On robots.txt, Perplexity says: “Webmasters can use the following robots.txt tags to manage how their sites and content interact with Perplexity.” Perplexity publishes the IP addresses for PerplexityBot as a JSON file. On search, Perplexity says: “PerplexityBot is designed to surface and link websites in search results on Perplexity.” Cite Files requests this agent when it scans a site. Checked against Perplexity's documentation on 4 October 2026. --- ## Perplexity-User: user agent and how to block it Source: https://citefiles.com/crawlers/perplexity-user _Perplexity's user-triggered fetcher: its published user agent, what Perplexity says about robots.txt, and how to verify requests._ Perplexity-User is an entry in Perplexity's crawler documentation, listed here as user-triggered. Perplexity describes it this way: “Perplexity-User supports user actions within Perplexity. When users ask Perplexity a question, it might visit a web page to help provide an accurate answer and include a link to the page in its response.” Its user-agent string as Perplexity publishes it is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user). On robots.txt, Perplexity says: “Since a user requested the fetch, this fetcher generally ignores robots.txt rules.” Perplexity publishes the IP addresses for Perplexity-User as a JSON file. Cite Files does not request this agent when it scans a site. Checked against Perplexity's documentation on 4 October 2026. --- ## Google-Extended: robots.txt token and blocking it Source: https://citefiles.com/crawlers/google-extended _Google's control token: its robots.txt token, what Google says about robots.txt, and how to block it._ Google-Extended is an entry in Google's crawler documentation, listed here as control token. Google describes it this way: “Google-Extended is a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models” Google says: “Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity.” On search, Google says: “Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.” Cite Files does not request this agent when it scans a site. Checked against Google's documentation on 4 October 2026. --- ## Applebot: user agent and how to block it Source: https://citefiles.com/crawlers/applebot _Apple's crawler: its published user agent, what Apple says about robots.txt, and how to verify requests._ Applebot is an entry in Apple's crawler documentation. Apple says the data Applebot crawls may also be used for training, and that disallowing Applebot-Extended opts out of that use. Apple describes it this way: “The data crawled by Applebot is used to power various features, such as the search technology integrated into many user experiences in Appleʼs ecosystem including Spotlight, Siri, and Safari. Enabling Applebot in robots.txt allows website content to appear in search results for Apple users around the world in these products.” Its user-agent string as Apple publishes it is Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot). On robots.txt, Apple says: “Applebot respects standard robots.txt directives in general search crawls that are targeted at Applebot.” Apple says traffic from Applebot is generally identified by reverse DNS in the *.applebot.apple.com domain, or by matching the IP address against the CIDR prefixes in its published JSON file. On search, Apple says: “Enabling Applebot in robots.txt allows website content to appear in search results for Apple users around the world in these products.” Cite Files does not request this agent when it scans a site. Checked against Apple's documentation on 4 October 2026. --- ## Applebot-Extended: robots.txt token and blocking it Source: https://citefiles.com/crawlers/applebot-extended _Apple's control token: its robots.txt token, what Apple says about robots.txt, and how to block it._ Applebot-Extended is an entry in Apple's crawler documentation, listed here as control token. Apple describes it this way: “Web publishers can opt-out from having their content used to train generative foundation models by disallowing Applebot-Extended in the robots.txt file.” Apple says: “Applebot-Extended does not crawl webpages.” It is a robots.txt token, so no request carries it. On search, Apple says: “Webpages that disallow Applebot-Extended can still be included in search results.” Cite Files does not request this agent when it scans a site. Checked against Apple's documentation on 4 October 2026. --- ## CCBot: user agent and how to block it Source: https://citefiles.com/crawlers/ccbot _Common Crawl's archive crawler: its published user agent, what Common Crawl says about robots.txt, and how to verify requests._ CCBot is an entry in Common Crawl's crawler documentation, listed here as archive. Common Crawl describes it this way: “Common Crawl is a non-profit foundation founded with the goal of democratizing access to web information by producing and maintaining an open repository of web crawl data that is universally accessible and analyzable by anyone.” Its user-agent string as Common Crawl publishes it is CCBot/2.0 (https://commoncrawl.org/faq/). On robots.txt, Common Crawl says: “To prevent Common Crawl from crawling your website, include the following in your robots.txt” Common Crawl says CCBot runs on dedicated IP address ranges with reverse DNS (except over IPv6, where reverse DNS is not yet supported), and publishes the ranges as JSON. Cite Files does not request this agent when it scans a site. Checked against Common Crawl's documentation on 4 October 2026. --- ## meta-externalagent: user agent and how to block it Source: https://citefiles.com/crawlers/meta-externalagent _Meta's crawler: its published user agent, what Meta says about robots.txt, and how to block it._ meta-externalagent is an entry in Meta's crawler documentation. Meta’s page says it crawls the web “for use cases such as training foundation AI models or improving products by indexing content directly”, so “training” is one use it names, not the only one, and the kind shown here is our shorthand. Meta describes it this way: “The Meta-ExternalAgent crawler crawls the web for use cases such as training foundation AI models or improving products by indexing content directly.” Its user-agent string as Meta publishes it is meta-externalagent/1.1 (+/documentation/sharing/webmasters/web-crawlers). On robots.txt, Meta says: “By configuring the robots.txt file on your website, you can specify to the Meta web crawlers how you would prefer them to interact with your site.” Cite Files does not request this agent when it scans a site. Checked against Meta's documentation on 4 October 2026. --- ## Bingbot: user agent and how to block it Source: https://citefiles.com/crawlers/bingbot _Bing's crawler: its published user agent, what Bing says about robots.txt, and how to verify requests._ Bingbot is an entry in Bing's crawler documentation. Bing’s page calls Bingbot its “standard crawler” and does not label a purpose, so we list it as “search” on our own classification, which is not a vendor statement. Bing describes it this way: “Bingbot is our standard crawler and handles most of our crawling needs each day.” Its user-agent string as Bing publishes it is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36. On robots.txt, Bing says: “Robots.txt files can be configured to tell Bing crawlers how to interact with your website.” Bing says user agent strings are easy to spoof and points to its Verify Bingbot tool, which checks whether an IP address belongs to Bingbot. On crawl rate, Bing says: “Bing Webmaster Tools allow you to control crawl rates by the hour using the Crawl control tool.” Cite Files does not request this agent when it scans a site. Checked against Bing's documentation on 4 October 2026. --- ## Guides to llms.txt, ai.txt and AI crawler access Source: https://citefiles.com/guides _Five sourced explainers: llms.txt, ai.txt, Content-Signal, GPTBot against OAI-SearchBot, and CDN blocks that robots.txt does not show._ Plain explainers for the terms people search when they ask whether an AI assistant can read their site, each ending in the free tool that checks it. There are five. The llms.txt guide sets out the proposal's structure and what the file does not do. The ai.txt guide separates training terms from crawler access. The Content-Signal guide explains the three keys, search, ai-input and ai-train, and where our own advice and Lighthouse disagree. The GPTBot guide separates OpenAI's training crawler, its ChatGPT search crawler and its user-triggered fetcher. The last guide explains how a CDN or firewall can refuse AI crawlers even when robots.txt allows them. Every claim about a standard or a vendor links the primary source: the standard itself or the vendor's own documentation. Claims about how Cite Files measures something are checked against its code and its methodology page. --- ## What is llms.txt? Format, examples and limits Source: https://citefiles.com/guides/llms-txt _What the llms.txt proposal specifies, how to structure one, why a site can serve an HTML page as the file, and what the file does not do._ llms.txt is a proposal for a markdown file at the root of a website that gives an AI assistant a short, curated map of the site. It is not a standard, and nothing obliges any crawler to fetch it. The proposal specifies one H1 with the site name, which is the only required part, then a blockquote summary, optional prose, and H2 sections holding lists of links, each with an optional note. A section called Optional marks links an agent can skip when it needs less. The file can sit at the root or at a subpath, where it covers the pages under that path. The guide also covers llms-full.txt, which the proposal text does not define, and the HTML-shell trap, where a server answers the path with a normal page and a 200 status so a naive checker reports a file that does not exist. It ends with the free validator and generator. --- ## What is ai.txt, and how is it different from robots.txt? Source: https://citefiles.com/guides/ai-txt _ai.txt states terms for AI training use, robots.txt asks crawlers to stay out of paths. What each file says, what neither enforces, and the alternative._ ai.txt is a text file at the root of a website that states whether its content may be used to train AI models. robots.txt answers a different question, which is which crawlers are asked to fetch which paths, and RFC 9309 says its rules are not a form of access authorization. A site can allow a crawler to read its pages while reserving training use. That is a coherent position, and it is the reason the two files are separate. Neither enforces anything. ai.txt is a convention published by Spawning, not a ratified standard, and it records intent. The guide also covers the robots.txt-native alternative, a Content-Signal line, and why a site's ai.txt and robots.txt should state the same position. It ends with the free ai.txt generator, which writes an ai.txt from the position your robots.txt already takes. --- ## Content-Signal in robots.txt: search, ai-input, ai-train Source: https://citefiles.com/guides/content-signal _The three Content-Signal keys as contentsignals.org defines them, where the line goes in robots.txt, and why Lighthouse 12.x and our advice disagree._ Content-Signal is a line that can be added to robots.txt to state how content may be used once a crawler has it. contentsignals.org defines three keys, each set to yes or no: search, for building a search index and returning results; ai-input, for inputting content into AI models, such as retrieval augmented generation; and ai-train, for training or fine-tuning. It states a preference and blocks nothing. The same site notes that not all automated systems honour robots.txt and some may ignore the directive. The guide covers where the line goes, with a copy-paste example, and a disagreement between tools that Cite Files documents: Lighthouse 12.x, the version our technical audit runs, flags the line as an unknown directive (13.0.2 and later accept it), and removing it from this site's file moved the Lighthouse 12.x SEO score from 92 to 100 without changing what a crawler does. It ends with the robots.txt tester. --- ## GPTBot vs OAI-SearchBot vs ChatGPT-User Source: https://citefiles.com/guides/gptbot-vs-oai-searchbot _OpenAI documents three agents with three purposes: training, ChatGPT search and user-triggered fetches. What blocking each does, with robots.txt snippets._ OpenAI documents separate agents for separate purposes. GPTBot crawls content that may be used in training its generative models. OAI-SearchBot surfaces sites in ChatGPT search results. ChatGPT-User fetches pages when a person asks ChatGPT or a custom GPT a question. OpenAI says each robots.txt setting is independent of the others, so a site can allow OAI-SearchBot and disallow GPTBot. It also says robots.txt rules may not apply to ChatGPT-User, because those requests are initiated by a user. The guide quotes OpenAI's page for what blocking each one does, gives copy-paste robots.txt snippets for three common choices, and explains why a Disallow under the wildcard group does not reach a crawler that has its own group. It ends with the AI crawler access checker and the robots.txt tester. --- ## Why AI crawlers get blocked when robots.txt allows them Source: https://citefiles.com/guides/ai-crawler-blocked-by-cdn _A CDN or firewall can refuse AI crawlers whatever robots.txt says. How a two-request comparison detects it, and what that test cannot prove._ A site can publish a robots.txt that allows every AI crawler and still refuse them, because robots.txt is a request read by the crawler while a CDN or firewall rule decides at the edge, before the site sees the request. RFC 9309 says robots.txt rules are not a form of access authorization. A scanner cannot read edge rules, so it asks: one request as a desktop browser, then one as each crawler, compared by status, response size and whether the page title survived. A refusal status is reported as blocked and a bot-check page as challenged. The guide explains what the test cannot prove. Requests come from the scanner's own address, not the vendor's network, so a rule that checks the crawler's address can refuse the scanner and still serve the real crawler. It ends with the AI crawler access checker. ---