CiteFilesBot
If you are reading this because you found us in your logs, here is exactly what we were doing and how to stop us.
What it is
CiteFilesBot visits a site for one of two reasons. Somebody asked. A person — usually the site’s owner — ran a scan, used one of the free tools, asked an AI agent to use our MCP server, or set up a weekly watch. Those visits identify as CiteFilesBot/1.0 (+https://citefiles.com/bot; audit requested by a visitor).
Our published sample. From 3 November 2026, CiteFilesBot will also measure a fixed, dated list of about a thousand well-known sites, on a schedule, so that the open record describes more than the sites people chose to scan. The list is a fixed list of well-known sites, published on the record when the first edition starts, minus the exclusions published with it, and each site will be measured about once a month. Those visits will identify as CiteFilesBot/1.0 (+https://citefiles.com/bot; published sample). Until that date every visit is one somebody asked for.
When somebody asks
A scan finds your site’s address with one request for your homepage, then reads your robots.txt before anything else, and stops there if it disallows us from your homepage (below). Otherwise it reads your sitemap, twelve conventional file paths (the list is on the methodology page), up to twenty-five pages, and one deliberately impossible address, so a real file can be told apart from a page your server returns for everything. It requests none of the paths your robots.txt disallows for CiteFilesBot, with one exception: the crawler-access test asks for your homepage as a browser and as each AI crawler, and follows your homepage’s own redirect within your site wherever it leads, because the page a visitor lands on is the thing being measured. That is the whole scan, usually well under a minute.
Two optional extras, which the scan’s owner has to ask for, also identify as CiteFilesBot, and each reads your robots.txt first: the technical checks use the copy the scan read if it is under a day old and read it afresh otherwise, and the “Check your published files” button reads it afresh every time. If it cannot be read, both treat that as a disallow. The technical checks look at up to 40 of the links on your own site that your homepage points to, skipping any your robots.txt disallows for CiteFilesBot. Each link gets a HEAD request; if your server answers 405 or 501, or does not answer, we send one GET before calling the link broken. The button reads robots.txt and then requests five fixed paths once each with a GET, /llms.txt, /llms-full.txt, /ai.txt, /robots.txt and /AGENTS.md, to see whether the files we generated are live and correctly served. It skips any of those paths your robots.txt disallows for CiteFilesBot, except /robots.txt itself, which it always requests. Both follow up to five redirects, other than into a path your robots.txt disallows for CiteFilesBot.
The full report and the technical checks also open a few of the pages the scan already read in a real browser, at most four for the report and three for the technical checks, to see what JavaScript adds and to run Lighthouse and an accessibility check. A browser loads what those pages load, so those visits also fetch their scripts, styles and images. The technical checks skip a page your robots.txt now disallows for CiteFilesBot.
A scheduled visit
From 3 November 2026, each visit to a site in our published sample will start with robots.txt. Before a scheduled visit we will request your robots.txt as CiteFilesBot, and if it disallows CiteFilesBot from your homepage — in a group for CiteFilesBot, or in the wildcard group when there is none — we will make no further request and will not measure the site. If robots.txt answers with a server error or does not answer, we will treat that as a disallow, as RFC 9309 asks; if it asks us to slow down (429), we will stop; either way we will try another day. We will read it fresh every time.
After that we will request: your homepage once as CiteFilesBot to follow your own redirect (and robots.txt again if the redirect lands on a different host name); your homepage once as a desktop browser and once as each of the four crawlers described below; and /llms.txt plus one deliberately impossible address, so a real file can be told apart from a page your server returns for everything. That will be nine requests when nothing redirects, and at most 24 counting redirects, one after another. If robots.txt sets a Crawl-delay for CiteFilesBot we will wait that long between requests, up to ten seconds.
Within three days a second server of ours, on a different network address, will repeat the robots.txt check, the browser request and the four crawler requests. We will publish a result about a site only when both servers got the same answer.
How to recognise it
When somebody asked us to look at a site:
CiteFilesBot/1.0 (+https://citefiles.com/bot; audit requested by a visitor)When the visit is part of our published sample (from 3 November 2026):
CiteFilesBot/1.0 (+https://citefiles.com/bot; published sample)One exception, and we would rather be upfront about it: the crawler-access test deliberately sends the user-agent strings OpenAI and Perplexity publish for GPTBot, OAI-SearchBot and PerplexityBot, and the string ClaudeBot is seen sending (Anthropic documents the token, not the full string). That is the only way to observe how an origin treats those agents, and it is the single most useful thing we tell site owners. The checker requests your public homepage as a browser and as each of those four agents, plus /robots.txt — nothing else. It follows a redirect within your own site and not to another site, makes at most 18 requests in total, and makes no attempt to defeat any challenge it receives. The measurement of our published sample will send the same four strings, from our servers rather than the vendors’ networks.
The single-purpose tools fetch less than a scan, and follow a redirect only within the same site. The llms.txt validator, the robots.txt tester and the ai.txt generator each request one file, /llms.txt or /robots.txt: one request, or at most three if your own site redirects. The validator and the tester request that file directly, without consulting robots.txt first, because it is a file published for crawlers. The AI crawler access checker requests the homepage as a browser and as each of the four agents above, plus /robots.txt, at most 18 requests in total. The llms.txt generator reads /robots.txt, the homepage, one sitemap and up to ten pages, and honours robots.txt before reading pages; if the sitemap is an index it takes pages from links on the homepage instead. Its hard cap is 20 requests, redirects included. A pasted input is never fetched.
How to block it
Add this to your robots.txt:
User-agent: CiteFilesBot
Disallow: /A scan somebody asks for requests your homepage once to find its address, then robots.txt. If robots.txt disallows CiteFilesBot from your homepage, the scan stops there and reports that it could not read the site, which is the honest outcome. If it disallows only particular paths, the scan does not request those paths (the crawler-access test still follows your homepage’s own redirect). If robots.txt answers with a server error or does not answer, the scan treats that as a disallow, as RFC 9309 asks, stops, and says so; if it asks us to slow down (429), the scan stops and says so too.
A site in our published sample will have its robots.txt re-checked at least every seven days; when we find the disallow we will stop measuring it and take it out of the record, its page and the downloadable data within seven days.
What we keep
Page titles, descriptions, headings, word counts, structured-data types, and a text excerpt used to generate the files a site owner asked for. We do not keep full page copies, we do not republish your content, and we do not train anything on it.
For a site in the published sample we will keep the measurement only: response codes and sizes, the homepage title, the first 4,000 characters of robots.txt, which crawlers were served or refused, and what the edge identified itself as. No page text. We will keep the latest three measurements of each site, plus any measurement a dated report cites. To be taken out, disallow CiteFilesBot (above) or write to [email protected] from an address at your domain — we will remove it within two working days, and we will not ask why. A dated report will keep its totals, say “withheld at the site’s request”, and drop the row from its download.
If you would like a scan of your site removed, write to [email protected] from an address at that domain and we will delete it.
Last reviewed . Maintained by Cite Files — corrections to [email protected].