What is llms.txt? Format, examples and limits

llms.txt is a proposal for a markdown file at the root of a website that gives an AI assistant a short, curated map of the site. It is not a standard, and nothing obliges any crawler to fetch it. This guide sets out what the file contains, how to write one, and the trap that makes a checker report a file that is not there.

What the proposal specifies

The llms.txt proposal asks for a markdown file named llms.txt at the root of a site, or at any subpath such as /docs/llms.txt. A file covers the URLs under its path, and where more than one applies, agents should use the most specific one. The proposal lists the contents in a fixed order:

• an optional byte-order mark;
• one H1 with the name of the project or site, which is the only required part;
• a blockquote with a short summary containing the key information needed to understand the rest;
• zero or more markdown sections of any type except headings, with more detail;
• zero or more sections opened by an H2, each holding a file list: a markdown list in which every item has a link, written [name](url), optionally followed by a colon and notes.

The proposal calls a section named Optional the place for secondary information, links an agent can skip when it needs a shorter context. A small file in this shape looks like this:

# Example Bakery

> A family bakery. The menu, opening hours and allergen information are linked below.

## Pages

- [Menu](https://example.com/menu): Current bread and pastry range, with allergens
- [Opening hours](https://example.com/hours): Hours by day, including bank holidays

## Optional

- [Our history](https://example.com/about): How the bakery started

What it is for, and what it is not

The proposal describes llms.txt as a curated overview, where a sitemap is a list of all the indexable pages. It also says robots.txt lets automated tools know what access is considered acceptable, while llms.txt is used on demand, when an agent needs information about a topic while assisting a user.

What it does not do is bind anyone. A crawler that never requests the file breaks no rule, and no tool on this site can show that publishing one changes whether you are cited. OpenAI’s crawler documentation and Perplexity’s crawler documentation each point readers to an llms.txt for their documentation index. That shows vendors publish the file for their own documentation. It says nothing about what their crawlers fetch from yours.

llms-full.txt

llms-full.txt is a companion name that the proposal text we cite does not define. Our tools treat it as the place for the full text of the pages an index file links to. Our validator flags an llms-full.txt that is very thin, has no markdown headings, leaks HTML markup or is over 5 MB. This site’s own /llms.txt and /llms-full.txt are generated from the same list of pages as its sitemap, so the three cannot drift apart by hand.

Where to put it

At the root, as /llms.txt, served as plain text. The proposal also allows a subpath, in which case the file covers the pages under it. For pages that have a clean markdown version, it suggests serving that version at the same URL with .md appended, and it recommends link relations so clients can find the files: rel="alternate" with the type text/markdown for the markdown version, and rel="describedby" for the llms.txt that covers the page. Those can be HTML link elements or an HTTP Link header.

The HTML-shell trap

A missing file can look present. Some sites answer 200 OK with their HTML page for any path they do not recognise, so a checker that treats 200 as found reports an llms.txt that was never there. Our scanner first requests a deliberately impossible path. If that returns 200, every later 200 is compared against the page it returned, and a file only counts as present when the body is a file rather than a page. You can check by hand: a real llms.txt starts with a hash and a title, not an angle bracket.

What the validator checks

Our validator reports a missing H1, more than one H1, a missing blockquote summary, no H2 sections, relative links, links without descriptions and bullets in a link list that are not links. Flagging relative links is our rule, made because a file is fetched without the page around it; the proposal does not state it. The score grades structure only. It does not request the links, so a file can score well and point at pages that no longer exist. Our methodology weighs crawler access above the file itself, because a perfect llms.txt is never read by a crawler the server turns away.

Questions