# Cite Files > Cite Files checks whether AI assistants can reach, read and cite a website — testing edge-level crawler blocking, robots.txt, sitemaps, llms.txt and ai.txt — then generates the missing files from the site's own published text. Most audits read robots.txt and stop. robots.txt is a request, not a control: a site can publish welcoming rules while its CDN returns a challenge to the same crawlers. Cite Files requests each site as a browser and again as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, then compares the answers. Scoring rubric v1.3.0, scanner v1.13.0. The full rubric is published at https://citefiles.com/methodology — no score is ever set by a language model. ## Start here - [Cite Files](https://citefiles.com/): Enter a website address and get a Citation Readiness score out of 100, free and without an account. - [AI citation readiness FAQ: llms.txt, ai.txt, AI crawlers](https://citefiles.com/faq): Plain answers about the Citation Readiness score and the Agent Commerce Readiness tier: what is measured, what is not, and how to check it yourself. - [Contact](https://citefiles.com/contact): How to reach us: general questions and corrections, security and removal, account problems. - [Report a problem or suggest something](https://citefiles.com/feedback): Tell us if a result is wrong about your site, if something is broken, or what to build next. ## Tools - [Free AI crawler and llms.txt tools](https://citefiles.com/tools): Five free single-purpose checks, no account: validate an llms.txt, generate one, test whether an AI crawler is blocked, test a robots.txt for AI agents, and build an ai.txt. - [llms.txt validator: check your llms.txt file](https://citefiles.com/tools/llms-txt-validator): Paste an llms.txt or enter a site address and get the file graded, with each problem listed. Free, no account. - [llms.txt generator: build one from your own pages](https://citefiles.com/tools/llms-txt-generator): Enter a site address and get an llms.txt built only from that site's own page titles and meta descriptions. Free, no account. - [Is GPTBot blocked on my site? AI crawler access checker](https://citefiles.com/tools/ai-crawler-checker): Enter a site address and see whether GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot are served or blocked, beside what robots.txt says. Free, no account. - [robots.txt tester for AI crawlers](https://citefiles.com/tools/robots-txt-ai-tester): Paste a robots.txt or enter a site address and see, agent by agent, whether each AI crawler is allowed or disallowed. Free, no account. - [ai.txt generator](https://citefiles.com/tools/ai-txt-generator): Paste a robots.txt or enter a site address and get an ai.txt that states the same position on AI training. Free, no account. ## How it works - [How the AI citation readiness score works](https://citefiles.com/methodology): The full rubric — every check, its weight, and how each one is measured — plus the Agent Commerce Readiness tiers, a separate unscored audit of whether an agent can transact with the site. - [What has changed in the scoring](https://citefiles.com/methodology/changelog): Every change to the rubric and scanner, dated, with an honest note on which way it moved scores. - [About Cite Files](https://citefiles.com/about): Why the project exists, and the three things it will not do. - [Which websites block AI crawlers? An open record](https://citefiles.com/index/crawler-access): An open record of how real websites treat GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. - [Which sites can an AI agent actually transact with](https://citefiles.com/index/agent-commerce): An open record of Agent Commerce Readiness tiers, T0 to T5 — whether an AI agent can read prices, call an API and obtain its own credentials — across sites whose owners chose to run the check. - [Agent Commerce Readiness: can an AI agent buy from you?](https://citefiles.com/agent-commerce-readiness): What Agent Commerce Readiness measures — eight read-only checks bundled into a tier, T0 to T5 — how to run it on a site you own, and how to check every predicate yourself with curl. - [MCP server and GitHub Action for AI readiness checks](https://citefiles.com/integrations): Connect the audit to a coding agent over MCP, or to CI so a change that makes a site less citable fails the build. - [Answer tracking — coming soon](https://citefiles.com/pro): What a paid tier would add, why it costs money, and what stays free. Not available yet. - [AI crawlers and robots.txt tokens: a sourced reference](https://citefiles.com/crawlers): One page per AI crawler and robots.txt token (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot, Applebot-Extended, CCBot, meta-externalagent, Bingbot), each sourced from the operator's own documentation. - [GPTBot: user agent and how to block it](https://citefiles.com/crawlers/gptbot): OpenAI's training crawler: its published user agent, what OpenAI says about robots.txt, and how to verify requests. - [OAI-SearchBot: user agent and how to block it](https://citefiles.com/crawlers/oai-searchbot): OpenAI's search crawler: its published user agent, what OpenAI says about robots.txt, and how to verify requests. - [ChatGPT-User: user agent and how to block it](https://citefiles.com/crawlers/chatgpt-user): OpenAI's user-triggered fetcher: its published user agent, what OpenAI says about robots.txt, and how to verify requests. - [ClaudeBot: robots.txt token and blocking it](https://citefiles.com/crawlers/claudebot): Anthropic's training crawler: its robots.txt token, what Anthropic says about robots.txt, and how to verify requests. - [Claude-User: robots.txt token and blocking it](https://citefiles.com/crawlers/claude-user): Anthropic's user-triggered fetcher: its robots.txt token, what Anthropic says about robots.txt, and how to verify requests. - [Claude-SearchBot: robots.txt token and blocking it](https://citefiles.com/crawlers/claude-searchbot): Anthropic's search crawler: its robots.txt token, what Anthropic says about robots.txt, and how to verify requests. - [PerplexityBot: user agent and how to block it](https://citefiles.com/crawlers/perplexitybot): Perplexity's search crawler: its published user agent, what Perplexity says about robots.txt, and how to verify requests. - [Perplexity-User: user agent and how to block it](https://citefiles.com/crawlers/perplexity-user): Perplexity's user-triggered fetcher: its published user agent, what Perplexity says about robots.txt, and how to verify requests. - [Google-Extended: robots.txt token and blocking it](https://citefiles.com/crawlers/google-extended): Google's control token: its robots.txt token, what Google says about robots.txt, and how to block it. - [Applebot: user agent and how to block it](https://citefiles.com/crawlers/applebot): Apple's crawler: its published user agent, what Apple says about robots.txt, and how to verify requests. - [Applebot-Extended: robots.txt token and blocking it](https://citefiles.com/crawlers/applebot-extended): Apple's control token: its robots.txt token, what Apple says about robots.txt, and how to block it. - [CCBot: user agent and how to block it](https://citefiles.com/crawlers/ccbot): Common Crawl's archive crawler: its published user agent, what Common Crawl says about robots.txt, and how to verify requests. - [meta-externalagent: user agent and how to block it](https://citefiles.com/crawlers/meta-externalagent): Meta's crawler: its published user agent, what Meta says about robots.txt, and how to block it. - [Bingbot: user agent and how to block it](https://citefiles.com/crawlers/bingbot): Bing's crawler: its published user agent, what Bing says about robots.txt, and how to verify requests. - [Guides to llms.txt, ai.txt and AI crawler access](https://citefiles.com/guides): Five sourced explainers: llms.txt, ai.txt, Content-Signal, GPTBot against OAI-SearchBot, and CDN blocks that robots.txt does not show. - [What is llms.txt? Format, examples and limits](https://citefiles.com/guides/llms-txt): What the llms.txt proposal specifies, how to structure one, why a site can serve an HTML page as the file, and what the file does not do. - [What is ai.txt, and how is it different from robots.txt?](https://citefiles.com/guides/ai-txt): ai.txt states terms for AI training use, robots.txt asks crawlers to stay out of paths. What each file says, what neither enforces, and the alternative. - [Content-Signal in robots.txt: search, ai-input, ai-train](https://citefiles.com/guides/content-signal): The three Content-Signal keys as contentsignals.org defines them, where the line goes in robots.txt, and why Lighthouse 12.x and our advice disagree. - [GPTBot vs OAI-SearchBot vs ChatGPT-User](https://citefiles.com/guides/gptbot-vs-oai-searchbot): OpenAI documents three agents with three purposes: training, ChatGPT search and user-triggered fetches. What blocking each does, with robots.txt snippets. - [Why AI crawlers get blocked when robots.txt allows them](https://citefiles.com/guides/ai-crawler-blocked-by-cdn): A CDN or firewall can refuse AI crawlers whatever robots.txt says. How a two-request comparison detects it, and what that test cannot prove. ## For crawler operators - [CiteFilesBot: what it fetches and how to block it](https://citefiles.com/bot): What our crawler fetches, its user-agent string, and how to block it. ## Optional - [Privacy](https://citefiles.com/privacy): What is stored, for how long, and how to have it deleted. - [Terms](https://citefiles.com/terms): Terms of use in plain language, including what the report does and does not promise. ## Notes for agents - Full text of every page above is available in one request at https://citefiles.com/llms-full.txt - Our position on training use of this content is at https://citefiles.com/ai.txt - What you can rely on here, and what we do not claim: https://citefiles.com/AGENTS.md - Our crawler and how to block it: https://citefiles.com/bot - Agent Commerce Readiness is a second, separate audit — a tier (T0 to T5), never a score — run only when a site's owner triggers it, making never more than 40 read-only requests. Explainer: https://citefiles.com/agent-commerce-readiness — questions and answers for both products: https://citefiles.com/faq - Report pages under /report/ are per-visitor and are excluded from crawling.