These are three of the crawlers that read your site on behalf of AI products. They are not interchangeable, and the most common mistake is treating them as one thing to allow or block together.
The distinction that actually matters is not which company sent the bot. It is what the fetch is for: training a future model, building a search index, or fetching a page right now because someone asked a question.
Blocking a training crawler keeps your content out of the next model. Blocking a search or user-fetch crawler keeps you out of today’s answers. People who wanted the first often did the second by accident.
The three jobs a crawler can be doing
Training
Collecting text for a future model. The payoff to you is diffuse and unattributable: a model that has read you may describe your product correctly without citing you. Blocking this has no effect on whether you appear in answers today.
Search indexing
Building the index that a product searches when it needs live facts. Block this and you are absent from the citations, because the retrieval step cannot find you.
User-initiated fetch
Someone pasted your URL, or asked a question and the product decided to open a specific page. This is the closest thing to a real visitor. Blocking it means a user who explicitly asked about your page gets told it cannot be read.
The agents, by vendor
Vendors add, rename and retire user agents, so treat this as a starting point and check the current documentation before committing a policy.
OpenAI
GPTBot— training collection.OAI-SearchBot— the search index behind ChatGPT’s live lookups.ChatGPT-User— a fetch triggered by something a user asked.
Anthropic
ClaudeBot— the current crawler.Claude-User,Claude-SearchBot— user-initiated and search fetches.anthropic-ai,Claude-Web— older names still seen in logs and in copied robots.txt files.
Perplexity
PerplexityBot— indexing for its answer engine.Perplexity-User— a fetch for a specific question.
Worth knowing about
Google-Extended— not a crawler at all. It is a token in robots.txt that controls whether content already crawled by Googlebot may be used for Gemini training. Blocking it does not affect Search.CCBot— Common Crawl. Not an AI company, but its archive feeds many training sets.Applebot-Extended,meta-externalagent— the equivalent controls for Apple and Meta.
A worked robots.txt
This allows the retrieval and user-fetch agents and declines training. It is one defensible position, not the only one.
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /Two things to understand before pasting it. Directives are matched per user agent, so a group you do not name falls through to your User-agent: * rules. And robots.txt is a request, not a control: named crawlers from large vendors honour it, anonymous scrapers do not, and nothing in a text file protects content that should not be public.
Checking what is actually reaching you
Your access log is the only source of truth about who is crawling. On this server:
grep -ioE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|PerplexityBot|CCBot' \
/var/log/nginx/access.log | sort | uniq -c | sort -rnIf a bot you believe you allowed does not appear at all, check robots.txt before concluding anything else. If one you blocked keeps appearing, check whether the user agent string changed.
Common questions
Should I block AI crawlers?
It depends what your content is for. If the content is the product and you sell access to it, blocking training crawlers is defensible. If your content exists to market something else, blocking removes you from the answers people now ask instead of searching, which is usually the more expensive choice.
Does blocking GPTBot remove me from ChatGPT?
Not from its live answers. GPTBot is the training collector; live lookups go through the search agent. Blocking GPTBot while allowing OAI-SearchBot keeps you citable today and out of the next training run, which is what most people actually want.
Does Google-Extended affect my Google rankings?
No. It governs whether Google may use already-crawled content for generative AI training. Googlebot and Search are unaffected, which is what makes it the least costly of these opt-outs.
Can I allow some pages and block others?
Yes, with normal path rules per user agent. A common split is allowing documentation and product pages while disallowing a members area or a large reference dataset you would rather not have absorbed wholesale.
Do these bots respect crawl-delay?
Inconsistently, and it is not part of the original robots standard. If a crawler is hitting you hard enough to matter, rate-limit it at the server or CDN rather than asking politely in a text file.
Where to go next
How does ChatGPT choose sources? covers what happens after the fetch succeeds. How to add llms.txt to WordPress is the adjacent convention.
Did this answer it? Tell us if not and we will fix the article.