AI companies run dozens of distinct web crawlers, each with a different job. Some collect training data, some build search indexes used to answer questions, and some fetch a page only when a user asks. Treating them all as "AI bots" hides the differences that actually matter for your traffic.
This reference lists the crawlers that most often appear in robots.txt discussions, what each is used for, and a sensible default decision. The single most important distinction is training versus search: blocking a training crawler does not remove you from search-style answers, and allowing training does not guarantee citation.
Training crawlers vs search crawlers
- Training crawlers gather text to train or fine-tune models. Blocking them affects a model's long-term knowledge, not live answers.
- Search crawlers build an index that assistants query at answer time. These are the ones most relevant to being cited or linked today.
- User-initiated agents fetch a specific page when someone asks an assistant to open it. They act like a browser on that user's behalf.
A reasonable default for a site that wants visibility is: allow search and user-initiated agents, and decide about training agents according to your policy. There is no universally correct choice — blocking training is a legitimate business decision.
The full list
| Crawler | Operator | Purpose | Default suggestion |
|---|---|---|---|
GPTBot | OpenAI | Model training collection | Allow or block by policy |
OAI-SearchBot | OpenAI | Search index for ChatGPT | Allow if you want search visibility |
ChatGPT-User | OpenAI | User-initiated page fetch | Allow |
ClaudeBot | Anthropic | Model training collection | Allow or block by policy |
Claude-Web | Anthropic | Web fetch / browsing | Allow |
anthropic-ai | Anthropic | Additional Anthropic agent | Allow unless you have a reason not to |
PerplexityBot | Perplexity | Indexes pages for cited answers | Allow if you want citations |
Perplexity-User | Perplexity | User-initiated page fetch | Allow |
Google-Extended | Controls Gemini / Vertex training use | Set explicitly by policy | |
CCBot | Common Crawl | Builds a public training dataset | Allow or block by policy |
Bytespider | ByteDance | Model training collection | Allow or block by policy |
Applebot-Extended | Apple | Apple model training opt-out | Set explicitly by policy |
meta-externalagent | Meta | Model training collection | Allow or block by policy |
Amazonbot | Amazon | Training / assistant features | Allow or block by policy |
cohere-ai | Cohere | Model training collection | Allow or block by policy |
YouBot | You.com | Search index | Allow if you want visibility |
Diffbot | Diffbot | Web data / knowledge graph | Allow unless bandwidth is a concern |
AI2Bot | Allen Institute for AI | Research crawling | Usually harmless to allow |
omgili | Webz.io | Data collection for training products | Allow or block by policy |
The ones that matter most for visibility
If you only pay attention to a few, make them these:
OAI-SearchBotandPerplexityBot— directly tied to search-style retrieval and citations. Blocking these is the most likely to reduce discoverability in answers.Google-Extended— widely misunderstood. It controls whether Google uses your content for certain AI training, not whether you appear in Google AI Overviews. Normal Googlebot crawling governs that. See Google-Extended explained.GPTBotandClaudeBot— the headline training crawlers. Your decision here is about training policy, not immediate citation.
Important nuances
Defaults are allow. In robots.txt, a crawler you do not mention is allowed. So a site with no AI-specific rules is not "hidden" — it is open to all these agents. A checker that reports a crawler as "not mentioned" is reporting the absence of a rule, not a block.
Names change. Vendors add, rename, and retire user-agent tokens. The list above reflects commonly discussed agents, but it is not exhaustive and not permanent. Re-audit periodically.
Agent strings are case-insensitive in theory. The robots.txt specification treats user-agent tokens case-insensitively, but real-world parsers vary. Use the exact published spelling.
Robots.txt is a convention, not a wall. It expresses your preference and is respected by major vendors, but it cannot technically stop a determined client. Treat it as policy, not security.
Crawling is not the same as citation. Being crawled makes you eligible. Whether an assistant cites you also depends on content clarity, authority, and the specifics of the query. See how to get cited in Google AI Overviews and how to get cited by Perplexity.
How to audit your own site
Fetch your live robots.txt and look for Disallow rules attached to any of these tokens. Then run the free check at https://ai.ynnadmusic.uk, which parses the live file and reports the access decision for 19 crawlers at once. For a broader review, use the 12-point AI visibility checklist.
Record your decisions in a short comment at the top of your robots.txt so future editors understand why each agent is allowed or blocked.
Frequently asked questions
Which AI crawler should I never block if I want citations?
There is no guaranteed-visible crawler, but OAI-SearchBot and PerplexityBot are the most directly tied to search-style retrieval and citations. Blocking them is more likely to reduce discoverability than blocking a pure training crawler.
Does blocking all AI crawlers hurt my Google rankings?
Not directly. Google's normal crawler is Googlebot, which is separate from the AI-specific agents listed here. AI Overviews draw on the regular Google index, so ordinary crawlability and indexing still matter.
Are these the only AI crawlers that exist?
No. This is a practical list of commonly encountered agents, not an exhaustive registry. New agents appear and old ones change. Treat any such list as a snapshot and re-check periodically.
Do I need a separate rule for each crawler?
Only if you want different behaviour for each. Because crawlers are allowed by default, you only need explicit rules for the ones you want to block or explicitly allow. Group agents with the same policy under one User-agent block with a single Disallow or Allow.