GPTBot vs OAI-SearchBot vs ChatGPT-User

OpenAI documents separate crawlers and user agents for separate purposes, and a robots.txt rule for one does not decide the others. This guide uses OpenAI’s own page to separate three of them, GPTBot, OAI-SearchBot and ChatGPT-User, and gives robots.txt snippets for the common choices.

Three agents, three purposes

OpenAI’s crawler page says OpenAI uses OAI-SearchBot and GPTBot robots.txt tags to let webmasters manage how their sites and content work with AI, and that each setting is independent of the others. In its words:

• GPTBot: “It is used to crawl content that may be used in training our generative AI foundation models. Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models.”
• OAI-SearchBot: “OAI-SearchBot is used to surface websites in search results in ChatGPT’s search features. Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.”
• ChatGPT-User: used for certain user actions in ChatGPT and Custom GPTs. “ChatGPT-User is not used for crawling the web in an automatic fashion. Because these actions are initiated by a user, robots.txt rules may not apply. ChatGPT-User is not used to determine whether content may appear in Search.”

The page lists a further agent, OAI-AdsBot, which validates pages submitted as ads on ChatGPT. It is outside this guide.

What blocking each one does

The effects below are as OpenAI describes them; we have not tested any of them.

• Disallow GPTBot: per OpenAI, this indicates your content should not be used in training its generative AI foundation models. Because each setting is independent, it is not an opt-out from ChatGPT search.
• Disallow OAI-SearchBot: your site is not shown in ChatGPT search answers, though it can still appear as a navigational link. OpenAI says it can take about 24 hours from a robots.txt update for its search systems to adjust.
• Disallow ChatGPT-User: OpenAI says robots.txt rules may not apply to it, and that OAI-SearchBot is the one to use in robots.txt for search opt-outs and automatic crawl. A Disallow for ChatGPT-User alone may not do what you expect.

OpenAI adds that if a site has allowed both GPTBot and OAI-SearchBot, it may use the results of just one crawl for both purposes to avoid duplicative crawling.

Which one to allow

Work from the outcome you want. To appear in ChatGPT search answers, allow OAI-SearchBot, which is the agent OpenAI names for that. To signal that your content should not be used in training, disallow GPTBot. The two do not depend on each other, which is why OpenAI’s own example allows one and disallows the other. To stop user-triggered fetches, robots.txt may not be enough, because OpenAI says its rules may not apply to ChatGPT-User. An edge rule is a different mechanism with different consequences, and the guide linked at the end of this page covers it.

Copy-paste robots.txt

Appear in ChatGPT search, opt out of training:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Opt out of both:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

Allow both:

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

These are requests, not controls. Under RFC 9309 a crawler obeys the group that matches its own product token, and uses the wildcard group only when no group matches. A Disallow under User-agent: * therefore does not reach a crawler that has a group of its own, and a rule meant for everyone has to be repeated inside each named group.

Finding them in your logs

For each of these agents OpenAI publishes an example user-agent string, and notes that the version number may change, so match on the agent name rather than the version. For GPTBot and OAI-SearchBot it adds that requests for robots.txt may carry an extra robots.txt marker in the string, so you can tell them from requests for other resources when your logs do not record paths. It also publishes a JSON list of IP addresses for each agent, which is the way to check that a request claiming to be one of them came from OpenAI.

Checking what your site does

The robots.txt tester evaluates the GPTBot, OAI-SearchBot and ChatGPT-User tokens and shows which rule decided each one. The AI crawler checker asks a different question: it requests your homepage as a browser and as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, and compares the answers. It does not send ChatGPT-User.

The checker sends the strings OpenAI publishes, from our own server and not from OpenAI’s network. OpenAI recommends allowing requests from its published IP ranges for OAI-SearchBot, so a site that checks the address could refuse the checker and still serve the real crawler. A passing result is one request at one moment. See why a crawler can be blocked even when robots.txt allows it.

Questions