Guides to llms.txt, ai.txt and AI crawler access

Plain explainers for the terms people search when they ask whether an AI assistant can read their site. Every claim about a standard or a vendor links the standard itself or the vendor’s own documentation, and each guide ends in the free tool that checks it.

• What is llms.txt? Format, examples and limits

  What the llms.txt proposal specifies, how to structure one, why a site can serve an HTML page as the file, and what the file does not do.

• What is ai.txt, and how is it different from robots.txt?

  ai.txt states terms for AI training use, robots.txt asks crawlers to stay out of paths. What each file says, what neither enforces, and the alternative.

• Content-Signal in robots.txt: search, ai-input, ai-train

  The three Content-Signal keys as contentsignals.org defines them, where the line goes in robots.txt, and why Lighthouse 12.x and our advice disagree.

• GPTBot vs OAI-SearchBot vs ChatGPT-User

  OpenAI documents three agents with three purposes: training, ChatGPT search and user-triggered fetches. What blocking each does, with robots.txt snippets.

• Why AI crawlers get blocked when robots.txt allows them

  A CDN or firewall can refuse AI crawlers whatever robots.txt says. How a two-request comparison detects it, and what that test cannot prove.

Claims about how Cite Files measures something are checked against its code and the methodology. The tools themselves are on the tools page, and the full check runs them all at once.

Last reviewed 4 October 2026. Maintained by Cite Files — corrections to [email protected].