What is ai.txt, and how is it different from robots.txt?

ai.txt is a text file at the root of a website that states whether the site’s content may be used to train AI models. robots.txt answers a different question, which is which crawlers are asked to fetch which paths. This guide explains the difference, what each file can and cannot do, and a way to keep the position in one file.

What ai.txt is

Spawning, the organisation behind the format, describes an ai.txt file as a simple text file placed in the root directory of a website that provides instructions on whether the images and text hosted on the domain can be used to train commercial AI models. Its page says to upload the file to / or /.well-known/ beside robots.txt, that it does not affect traditional search engine crawlers or a site’s SEO, and that the generator’s default is that all content is opted out.

Training terms versus crawler access

robots.txt, specified in RFC 9309, sets out rules that crawlers are requested to honour when accessing URIs, and the RFC is explicit that those rules are not a form of access authorization. ai.txt is about use: the terms on which content may be used for training once it has been obtained.

On our methodology page we call these two questions retrieval and training. Retrieval is whether an assistant may fetch your page now to answer a question and cite you. Training is whether your content may become part of a model. A site can allow the first and reserve the second, and that is a coherent position. What we flag is the contradiction: an ai.txt that grants training use while robots.txt disallows the crawlers from reading the site at all.

What ai.txt does not do

It enforces nothing. Spawning’s terms for its tool say the tool is designed to aid in communicating preferences about data use and that Spawning does not guarantee ai.txt will prevent unauthorized data usage. Its FAQ says that while many AI miners respect the rules, some may not. We print two caveats on every ai.txt result: it is a convention rather than a ratified standard, and adoption is partial.

Spawning’s page also says it passes the permissions in ai.txt files to a growing list of AI researchers and partners through its API, and that for everyone else the file is a machine-readable opt-out that can be parsed and respected. That is the publisher of the format describing its own reach. We have no independent source on who reads the file, so we do not claim any particular crawler does.

What it gives you is a record. A stated reservation is evidence of intent and an unstated one is not, which is the reason to publish one. It is not a reason to expect obedience. Refusing a crawler that ignores your files takes a rule at your CDN, firewall or server; the guide to edge blocks explains how that differs from robots.txt.

A robots.txt-native alternative

A site that would rather keep its position in one file can declare it in robots.txt with a Content-Signal line. contentsignals.org defines three keys, search, ai-input and ai-train, each set to yes or no, and describes the mechanism as Cloudflare’s implementation of a way to declare how automated systems should use content. Like ai.txt, it states a preference: the same page warns that not all automated systems honor robots.txt and some may ignore the directives. RFC 9309 itself defines only user-agent, allow and disallow lines and says crawlers MAY interpret other records. The Content-Signal guide covers the three keys.

Choosing what to publish

Decide the position first, then state it consistently. If your robots.txt already carries a Content-Signal line, an ai.txt that says the opposite helps nobody. Our generator reads your robots.txt and writes an ai.txt that expresses the same position, and says so in the file when the two disagree or when it is broader than your robots.txt, rather than choosing a licensing stance for you. If your robots.txt gives no training signal there is no position to copy, so the file starts from a template default, the reservation, with comments asking you to confirm it before you publish.

Once it is live, fetch /ai.txt and confirm it is a file. Our scanner treats a 200 response with an HTML body as a catch-all route, not an ai.txt, for the same reason it does for llms.txt: a server that answers every unknown path with its page layout makes a missing file look present.

This site follows its own advice. Its ai.txt permits retrieval and citation and, for its published prose, training. It excludes audit results for other people’s websites. Its robots.txt states the same position in a Content-Signal line, search=yes and ai-train=yes, and says nothing about ai-input.

Questions