GPTBot is OpenAI's web crawler for collecting training data. Whether you allow or block it is controlled entirely by one line in your robots.txt — but OpenAI runs three separate user-agents, and treating them as one is the most common mistake. Here is how each works and the exact rules to use.
The three OpenAI user-agents
| User-agent | Purpose | Blocking effect |
|---|---|---|
GPTBot | Collects pages for model training | Your content is excluded from future training crawls |
OAI-SearchBot | Crawls for ChatGPT's search index | Your pages may not surface in search-style answers |
ChatGPT-User | Fetches a page when a user explicitly asks ChatGPT to visit it | Can prevent on-demand visits to a specific URL |
These are independent. Blocking GPTBot does not block OAI-SearchBot, and vice versa. Decide about each one deliberately.
OpenAI documents that OAI-SearchBot is used for search indexing and is separate from training. The practical implication: if your goal is to be found and cited in ChatGPT's search-style answers, blocking OAI-SearchBot is the risky choice. If your goal is only to avoid training use, block GPTBot and leave the others alone.
Allow GPTBot
To welcome every OpenAI crawler, you can omit any rule for them (they are allowed by default), or be explicit:
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
Explicit rules are useful because they document your intent and survive future edits to the file.
Block GPTBot only
This is the most common configuration for sites that want search visibility but want to opt out of training collection:
User-agent: GPTBot
Disallow: /
Leave OAI-SearchBot and ChatGPT-User unmentioned so they remain allowed.
Block all OpenAI crawlers
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
Be aware that this closes all three paths, including search-style discovery and explicit user visits.
Common mistakes
Wrong user-agent casing or spelling. The documented token is GPTBot. Robots.txt user-agent matching is case-insensitive in the formal specification, but some real-world parsers are sloppy. Use the exact published spelling and avoid invented variants like GPT-Bot.
Rules grouped under the wrong agent. A Disallow line applies to the nearest preceding User-agent block. Putting Disallow: / under a general User-agent: * block and then adding an Allow for GPTBot below it can produce surprising results depending on rule specificity.
Allow after Disallow specificity. Robots.txt uses the most specific matching path, not strictly the last rule. For a given user-agent, a longer, more specific path wins over a broad one; where an Allow and a Disallow are equally specific, the Allow generally wins. Keep your rules unambiguous by using complete paths and avoiding overlapping patterns.
Forgetting the general crawler. If User-agent: * has Disallow: /, almost everything is blocked regardless of your AI-specific rules. AI crawlers typically respect the general block too.
A stale file behind a CDN or cache. You may edit the file but a cached copy keeps being served. Check the live response, not your local copy.
How platforms change robots.txt
- Cloudflare: A page rule, worker, or firewall setting can intercept or rewrite
/robots.txt. Check the Cloudflare dashboard if your edits do not appear live. - WordPress: SEO plugins such as Yoast and Rank Math generate or append to the virtual robots.txt file. Look in the plugin's tools section rather than editing a physical file.
- Shopify: robots.txt is controlled by a
robots.txt.liquidtemplate. Editing it requires theme access. - Static hosts / Workers: The file is a plain asset in your deploy. Redeploy after changing it.
In every case, verify the final response that a crawler actually receives.
How to verify your rules
- Fetch
https://yoursite.com/robots.txtin a browser or withcurland read the exact text served. - Run the free check at https://ai.ynnadmusic.uk — it parses the live file and reports the access decision for
GPTBot,OAI-SearchBot,ChatGPT-Userand 16 other crawlers. - If you control server logs, look for requests from the documented user-agent tokens after a few days.
- Re-check after any site migration, plugin update, or CDN change, since these often regenerate the file.
For the full set of crawlers and what each does, see Every AI crawler in 2026. For the nuance around Google's separate training flag, read Google-Extended explained.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT immediately?
No. It stops future training collection. It does not erase data already collected, and it does not affect search-style retrieval handled by other user-agents.
Is GPTBot the same as Googlebot?
No. GPTBot belongs to OpenAI; Googlebot belongs to Google. They are controlled by separate rules in the same robots.txt file. Google's AI training flag is Google-Extended, covered in its own guide.
Can I allow search but block training?
Yes. That is the main reason the agents are separate: block GPTBot while leaving OAI-SearchBot and ChatGPT-User allowed. Verify with the checker that only the intended agent is blocked.
Do all AI companies respect robots.txt?
Major operators publish crawlers that they state respect robots.txt, but enforcement and behaviour are controlled by each vendor and can change. Robots.txt is a widely followed convention, not a technical access-control system. Do not treat it as a security boundary.