AI companies run dozens of distinct web crawlers, each with a different job. Some collect training data, some build search indexes used to answer questions, and some fetch a page only when a user asks. Treating them all as "AI bots" hides the differences that actually matter for your traffic.

This reference lists the crawlers that most often appear in robots.txt discussions, what each is used for, and a sensible default decision. The single most important distinction is training versus search: blocking a training crawler does not remove you from search-style answers, and allowing training does not guarantee citation.

Training crawlers vs search crawlers

  • Training crawlers gather text to train or fine-tune models. Blocking them affects a model's long-term knowledge, not live answers.
  • Search crawlers build an index that assistants query at answer time. These are the ones most relevant to being cited or linked today.
  • User-initiated agents fetch a specific page when someone asks an assistant to open it. They act like a browser on that user's behalf.

A reasonable default for a site that wants visibility is: allow search and user-initiated agents, and decide about training agents according to your policy. There is no universally correct choice — blocking training is a legitimate business decision.

The full list

CrawlerOperatorPurposeDefault suggestion
GPTBotOpenAIModel training collectionAllow or block by policy
OAI-SearchBotOpenAISearch index for ChatGPTAllow if you want search visibility
ChatGPT-UserOpenAIUser-initiated page fetchAllow
ClaudeBotAnthropicModel training collectionAllow or block by policy
Claude-WebAnthropicWeb fetch / browsingAllow
anthropic-aiAnthropicAdditional Anthropic agentAllow unless you have a reason not to
PerplexityBotPerplexityIndexes pages for cited answersAllow if you want citations
Perplexity-UserPerplexityUser-initiated page fetchAllow
Google-ExtendedGoogleControls Gemini / Vertex training useSet explicitly by policy
CCBotCommon CrawlBuilds a public training datasetAllow or block by policy
BytespiderByteDanceModel training collectionAllow or block by policy
Applebot-ExtendedAppleApple model training opt-outSet explicitly by policy
meta-externalagentMetaModel training collectionAllow or block by policy
AmazonbotAmazonTraining / assistant featuresAllow or block by policy
cohere-aiCohereModel training collectionAllow or block by policy
YouBotYou.comSearch indexAllow if you want visibility
DiffbotDiffbotWeb data / knowledge graphAllow unless bandwidth is a concern
AI2BotAllen Institute for AIResearch crawlingUsually harmless to allow
omgiliWebz.ioData collection for training productsAllow or block by policy

The ones that matter most for visibility

If you only pay attention to a few, make them these:

  • OAI-SearchBot and PerplexityBot — directly tied to search-style retrieval and citations. Blocking these is the most likely to reduce discoverability in answers.
  • Google-Extended — widely misunderstood. It controls whether Google uses your content for certain AI training, not whether you appear in Google AI Overviews. Normal Googlebot crawling governs that. See Google-Extended explained.
  • GPTBot and ClaudeBot — the headline training crawlers. Your decision here is about training policy, not immediate citation.

Important nuances

Defaults are allow. In robots.txt, a crawler you do not mention is allowed. So a site with no AI-specific rules is not "hidden" — it is open to all these agents. A checker that reports a crawler as "not mentioned" is reporting the absence of a rule, not a block.

Names change. Vendors add, rename, and retire user-agent tokens. The list above reflects commonly discussed agents, but it is not exhaustive and not permanent. Re-audit periodically.

Agent strings are case-insensitive in theory. The robots.txt specification treats user-agent tokens case-insensitively, but real-world parsers vary. Use the exact published spelling.

Robots.txt is a convention, not a wall. It expresses your preference and is respected by major vendors, but it cannot technically stop a determined client. Treat it as policy, not security.

Crawling is not the same as citation. Being crawled makes you eligible. Whether an assistant cites you also depends on content clarity, authority, and the specifics of the query. See how to get cited in Google AI Overviews and how to get cited by Perplexity.

How to audit your own site

Fetch your live robots.txt and look for Disallow rules attached to any of these tokens. Then run the free check at https://ai.ynnadmusic.uk, which parses the live file and reports the access decision for 19 crawlers at once. For a broader review, use the 12-point AI visibility checklist.

Record your decisions in a short comment at the top of your robots.txt so future editors understand why each agent is allowed or blocked.

Frequently asked questions

Which AI crawler should I never block if I want citations?

There is no guaranteed-visible crawler, but OAI-SearchBot and PerplexityBot are the most directly tied to search-style retrieval and citations. Blocking them is more likely to reduce discoverability than blocking a pure training crawler.

Does blocking all AI crawlers hurt my Google rankings?

Not directly. Google's normal crawler is Googlebot, which is separate from the AI-specific agents listed here. AI Overviews draw on the regular Google index, so ordinary crawlability and indexing still matter.

Are these the only AI crawlers that exist?

No. This is a practical list of commonly encountered agents, not an exhaustive registry. New agents appear and old ones change. Treat any such list as a snapshot and re-check periodically.

Do I need a separate rule for each crawler?

Only if you want different behaviour for each. Because crawlers are allowed by default, you only need explicit rules for the ones you want to block or explicitly allow. Group agents with the same policy under one User-agent block with a single Disallow or Allow.