Glossary · How AI reads content
GPTBot
OpenAI's web crawler, which reads public pages to help train its models; sites can allow or block it in robots.txt.
# Allow OpenAI's crawlerUser-agent: GPTBotAllow: /# …or block it entirelyUser-agent: GPTBotDisallow: /
GPTBot is OpenAI’s web crawler. Announced in August 2023, it reads publicly available pages so their text can help improve future models — and, importantly, it respects the rules you set in robots.txt.
What GPTBot is
GPTBot is the bot OpenAI sends out to gather training data from the open web. It identifies itself with the user-agent string GPTBot, which is how you spot it in your server logs and how you address it in a crawl rule. Its job is narrow: collect publicly accessible text that may contribute to training future OpenAI models. It is not the mechanism ChatGPT uses to browse the live web while answering a question.
Training vs. browsing — two different bots
This is the distinction people most often get wrong. OpenAI runs separate crawlers for separate jobs, and they are controlled independently.
How to allow or block it
Control happens in robots.txt. Add a block addressed to GPTBot and choose your posture:
- Allow it.
User-agent: GPTBotfollowed byAllow: /lets it read your whole site. - Block it.
Disallow: /under the same user-agent keeps it out entirely. - Scope it. You can disallow specific paths while allowing the rest, the same way you would for any other crawler.
How to think about the decision
The allow-or-block call is a genuine judgment, not a default. Blocking GPTBot means your content won’t be used to train OpenAI’s models — a reasonable choice if that is your concern. But make it deliberately, and keep it separate from your decision about ChatGPT Search: if you want to appear in ChatGPT’s answers, the bot to leave unblocked is OAI-SearchBot, not GPTBot. Confusing the two is how sites accidentally cut themselves out of the answers they meant to be in.
Frequently asked
- What does GPTBot do?
- GPTBot is OpenAI's web crawler. It reads publicly available pages so that the text can help improve future OpenAI models. It identifies itself with the user-agent GPTBot and respects robots.txt, so site owners can decide whether to let it in.
- How do I block or allow GPTBot?
- Through robots.txt. Add a rule for User-agent: GPTBot and set Allow: / to permit it or Disallow: / to block it. Because GPTBot honors robots.txt, that single directive is the standard way to control it — no plugin or account setting required.
- Is blocking GPTBot the same as hiding from ChatGPT Search?
- No. GPTBot is about training. ChatGPT's live browsing uses a separate crawler, OAI-SearchBot, to fetch current sources when it answers. Blocking GPTBot keeps your content out of training but does not by itself remove you from ChatGPT Search — those are two different bots governed by two different robots.txt rules.