OpenAI · training
GPTBot: user agent and how to block it
Everything on this page is taken from OpenAI's own documentation, quoted and linked, and was checked on 4 October 2026. Where the documentation is silent the page says so instead of filling the gap.
What it is
Operator: OpenAI. Kind, as listed here: training.
“GPTBot is used to make our generative AI foundation models more useful and safe. It is used to crawl content that may be used in training our generative AI foundation models. Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models.”
Source: OpenAI: crawlers and user agents, checked 4 October 2026.
User-agent string
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
OpenAI labels this an example string and says the version number may change.
Source: OpenAI: crawlers and user agents. The robots.txt token to match is GPTBot.
Does it obey robots.txt?
“OpenAI uses OAI-SearchBot and GPTBot robots.txt tags to enable webmasters to manage how their sites and content work with AI.”
Source: OpenAI: crawlers and user agents, checked 4 October 2026.
Does it affect search?
“Each setting is independent of the others – for example, a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training OpenAI’s generative AI foundation models.”
Source: OpenAI: crawlers and user agents, checked 4 October 2026.
What else OpenAI says
“Example user-agent string (the version number may change)”
Source: OpenAI: crawlers and user agents, checked 4 October 2026.
robots.txt snippets
Two snippets that show the syntax for this token. They are examples, not advice: whether to allow or refuse GPTBot is your decision, and the effects above are the vendor's description of them.
Allow GPTBot
User-agent: GPTBot Allow: /
Disallow GPTBot
User-agent: GPTBot Disallow: /
How to verify requests
OpenAI publishes the IP addresses for GPTBot as a JSON file.
https://openai.com/gptbot.json
Source: OpenAI: crawlers and user agents.
What we measure
GPTBot is one of the four agents the Cite Files scanner requests with when it scans a site. Run the AI crawler checker on your own address to see how your site answers it, or read the open record of how scanned sites treat it.
Questions
What is GPTBot?
OpenAI describes GPTBot like this: “GPTBot is used to make our generative AI foundation models more useful and safe. It is used to crawl content that may be used in training our generative AI foundation models. Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models.”
Does GPTBot obey robots.txt?
The OpenAI documentation says: “OpenAI uses OAI-SearchBot and GPTBot robots.txt tags to enable webmasters to manage how their sites and content work with AI.”
How do I block GPTBot in robots.txt?
Add a group to robots.txt with the line “User-agent: GPTBot” followed by “Disallow: /”. That is an example of the syntax, not a recommendation. On search, OpenAI says: “Each setting is independent of the others – for example, a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training OpenAI’s generative AI foundation models.”
Also see the robots.txt tester for AI crawlers and the other entries in the AI crawler reference.
Last reviewed . Maintained by Cite Files — corrections to [email protected].