Content-Signal in robots.txt: search, ai-input, ai-train

Content-Signal is a line you can add to robots.txt to say how your content may be used once a crawler has it: for a search index, as input to an AI answer, or for training. It states a preference and blocks nothing. This guide covers the three keys as contentsignals.org defines them, where the line goes, and a disagreement between tools that we measured on our own site.

The three keys

The Content Signals site says the Content-Signal directive works by signalling a preference of either allowing (yes) or disallowing (no) certain categories of AI actions. It defines three, in its own words:

• search: “Building a search index and providing search results (e.g., returning hyperlinks and short excerpts from your website’s contents). Search does not include providing AI-generated search summaries.”
• ai-input: “Inputting content into one or more AI models (e.g., retrieval augmented generation, grounding, or other real-time taking of content for generative AI search answers).”
• ai-train: “Training or fine-tuning AI models.”

Two details matter when you choose values. The search definition excludes AI-generated search summaries, and the ai-input definition names generative AI search answers. In our own terms, ai-input is the key nearest to retrieval, an assistant taking your page to answer a question now, and ai-train is the training question.

The same site frames this as the IETF’s proposed AI Preferences work, called aipref, and says the vocabulary comes from a proposed standard drafted by the aipref working group. It is a proposal, not part of RFC 9309.

What the line looks like

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

That asks that your content be used for search and as AI input but not for training. The contentsignals.org examples place the line inside a user-agent group, and show rules for specific bots by giving them a group of their own, for instance one each for googlebot, bingbot and OAI-Searchbot:

User-Agent: googlebot
Content-Signal: ai-train=no, search=yes, ai-input=no
Allow: /

User-Agent: bingbot
Content-Signal: ai-train=no, search=yes, ai-input=no
Allow: /

User-Agent: OAI-Searchbot
Content-Signal: ai-train=no, search=yes, ai-input=no
Allow: /

The Content Signals site’s own robots.txt, when we read it, declares ai-train=yes, search=yes and ai-input=yes under a wildcard group, with a comment saying the directives use vocabulary from the proposed IETF standard. Our own robots.txt puts the line inside its wildcard group rather than above the first group. RFC 9309 defines only user-agent, allow and disallow lines, and says crawlers MAY interpret other records, so any use of the line is up to the crawler.

What it does not do

It is a statement, not a control. The Content Signals site notes that not all automated systems honor robots.txt, that some may ignore these directives, and that courts and regulators may conclude robots.txt files do not impose enforceable legal obligations. RFC 9309 says the same of the protocol it specifies: its rules are not a form of access authorization. We do not claim to know which crawlers read a Content-Signal line, and nothing on this site tests that.

Our tools read it in limited ways. The ai.txt generator reads the ai-train entry as a training signal, but a Disallow of a training crawler takes precedence over it, and an ai-train=yes that disagrees with those rules is flagged as a conflict in the ai.txt, which then follows the Disallow. The robots.txt tester gives an allow or disallow for each crawler token, read the way RFC 9309 describes, and does not evaluate Content-Signal.

Where our tools disagree

We recommend declaring Content-Signal in robots.txt, and Lighthouse 12.x, the version our technical audit runs, flags the line as an unknown directive. Both are true at once, so rather than pick a side we measured it, and recorded the result under “Where our tools disagree” on the methodology page. With Lighthouse 12.x, removing that single line takes this site’s Lighthouse SEO score from 92 to 100 and changes nothing about what a crawler does.

That version’s robots.txt parser predates the directive. Lighthouse 13.0.2 and later accept it: its robots.txt audit added content-signal to the directives it recognises in January 2026 (pull request 16767), and 13.0.1 does not have the change. We keep the line and accept the eight points on 12.x, because the declaration is the point and an SEO sub-score is a proxy. If you add the line and your Lighthouse number moves, check which version produced it; our technical report flags the line, so you are not left to discover the difference yourself. Whether you keep the line is your decision, and it is a trade, not an error on either side.

Check your file

Paste your robots.txt into the tester below to confirm that the allow and disallow lines say what you meant for each AI crawler token, and read the Content-Signal line yourself against the three definitions above. Check where the line sits too: the examples on contentsignals.org put it inside a user-agent group.

Questions