AI crawlers and robots.txt tokens: a sourced reference
One page for each crawler or token below. Every statement on those pages is quoted from the operator's own documentation and linked to it, and where the documentation says nothing the page says so.
Training, search, user-triggered and control tokens
The operators describe their agents in different groups. Some crawl content that may be used to train models, some index pages for a search feature, and some fetch a single page because a person asked an assistant a question. A few entries are robots.txt tokens only: they name a rule the operator reads, and no request carries them. A single operator can run several of these, each with its own token; OpenAI and Perplexity both say each of their robots.txt settings works independently of the others. The kind shown for each entry follows the operator's description; where a page marks a classification as ours rather than the operator's, it says so.
The 14 entries
| Token | Operator | Kind | User agent published? | robots.txt, per the vendor |
|---|---|---|---|---|
| GPTBot | OpenAI | training | Yes | Vendor documents a robots.txt rule |
| OAI-SearchBot | OpenAI | search | Yes | Vendor documents a robots.txt rule |
| ChatGPT-User | OpenAI | user-triggered | Yes | Vendor says rules may not apply |
| ClaudeBot | Anthropic | training | No | Vendor documents a robots.txt rule |
| Claude-User | Anthropic | user-triggered | No | Vendor documents a robots.txt rule |
| Claude-SearchBot | Anthropic | search | No | Vendor documents a robots.txt rule |
| PerplexityBot | Perplexity | search | Yes | Vendor documents a robots.txt rule |
| Perplexity-User | Perplexity | user-triggered | Yes | Vendor says rules may not apply |
| Google-Extended | control token | No | Token only: no requests | |
| Applebot | Apple | search and training | Yes | Vendor documents a robots.txt rule |
| Applebot-Extended | Apple | control token | No | Token only: no requests |
| CCBot | Common Crawl | archive | Yes | Vendor documents a robots.txt rule |
| meta-externalagent | Meta | training | Yes | Vendor documents a robots.txt rule |
| Bingbot | Bing | search | Yes | Vendor documents a robots.txt rule |
Sources, one per operator: OpenAI: crawlers and user agents · Anthropic: how to block the crawler · Perplexity crawlers · Google common crawlers · About Applebot · Common Crawl: CCBot · Meta web crawlers · Bing crawlers.
Two example robots.txt groups
Both are generated from the same data as the table. They are examples of the syntax, not a recommendation: whether to refuse any of these is your decision. Read each entry's page first: OpenAI and Perplexity say each of their settings is independent, and for ChatGPT-User and Perplexity-User the operators say robots.txt rules may not apply (OpenAI) or are generally ignored (Perplexity).
Tokens their operators link to AI model training (5)
# Example: the tokens their operators link to AI model training. User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: meta-externalagent Disallow: /
Applebot is not in this group. Apple says its data may also be used to help train Apple foundation models, but the training control Apple documents is Applebot-Extended, and Apple says enabling Applebot is what allows content to appear in search results for Apple users.
Every token on this page (14)
# Example: every token listed on this site. # Includes search crawlers: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot, Bingbot. # Their vendors say robots.txt rules may not apply to: ChatGPT-User, Perplexity-User. User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: ClaudeBot User-agent: Claude-User User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Google-Extended User-agent: Applebot User-agent: Applebot-Extended User-agent: CCBot User-agent: meta-externalagent User-agent: Bingbot Disallow: /
Check a real site
Paste a robots.txt into the robots.txt tester for AI crawlers to see which of these tokens it allows, or scan an address with the AI crawler checker to see how the server answers them.
Last reviewed . Maintained by Cite Files — corrections to [email protected].