{"id":28,"date":"2026-09-17T21:24:57","date_gmt":"2026-09-17T21:24:57","guid":{"rendered":"https:\/\/wpcontentwriter.com\/kb\/gptbot-vs-claudebot-vs-perplexitybot\/"},"modified":"2026-09-17T21:24:57","modified_gmt":"2026-09-17T21:24:57","slug":"gptbot-vs-claudebot-vs-perplexitybot","status":"publish","type":"post","link":"https:\/\/wpcontentwriter.com\/kb\/gptbot-vs-claudebot-vs-perplexitybot\/","title":{"rendered":"GPTBot vs ClaudeBot vs PerplexityBot"},"content":{"rendered":"<p>These are three of the crawlers that read your site on behalf of AI products. They are not interchangeable, and the most common mistake is treating them as one thing to allow or block together.<\/p>\n<p>The distinction that actually matters is not which company sent the bot. It is <strong>what the fetch is for<\/strong>: training a future model, building a search index, or fetching a page right now because someone asked a question.<\/p>\n<div class=\"kb-takeaway\">\n<span class=\"kb-takeaway__label\">The trade, in one line<\/span><\/p>\n<p>Blocking a training crawler keeps your content out of the next model. Blocking a search or user-fetch crawler keeps you out of today&#8217;s answers. People who wanted the first often did the second by accident.<\/p>\n<\/div>\n<h2>The three jobs a crawler can be doing<\/h2>\n<h3>Training<\/h3>\n<p>Collecting text for a future model. The payoff to you is diffuse and unattributable: a model that has read you may describe your product correctly without citing you. Blocking this has no effect on whether you appear in answers today.<\/p>\n<h3>Search indexing<\/h3>\n<p>Building the index that a product searches when it needs live facts. Block this and you are absent from the citations, because the retrieval step cannot find you.<\/p>\n<h3>User-initiated fetch<\/h3>\n<p>Someone pasted your URL, or asked a question and the product decided to open a specific page. This is the closest thing to a real visitor. Blocking it means a user who explicitly asked about your page gets told it cannot be read.<\/p>\n<figure class=\"kb-figure\">\n<svg viewBox=\"0 0 640 250\" role=\"img\" aria-label=\"Three crawler purposes and what blocking each one costs\">\n  <text class=\"fig-title\" x=\"10\" y=\"20\" font-size=\"14\">What you lose by blocking<\/text><\/p>\n<p>  <rect x=\"10\" y=\"38\" width=\"196\" height=\"94\" rx=\"10\" fill=\"#FFFFFF\" stroke=\"#E2E8F0\"\/>\n  <text x=\"26\" y=\"62\" font-size=\"12.5\" font-weight=\"600\">Training<\/text>\n  <text class=\"fig-muted\" x=\"26\" y=\"82\" font-size=\"10.5\">GPTBot, ClaudeBot,<\/text>\n  <text class=\"fig-muted\" x=\"26\" y=\"97\" font-size=\"10.5\">CCBot, Google-Extended<\/text>\n  <text x=\"26\" y=\"120\" font-size=\"11\">Cost: future recall<\/text><\/p>\n<p>  <rect x=\"222\" y=\"38\" width=\"196\" height=\"94\" rx=\"10\" fill=\"#FFFFFF\" stroke=\"#7C3AED\"\/>\n  <text x=\"238\" y=\"62\" font-size=\"12.5\" font-weight=\"600\" class=\"fig-accent\">Search index<\/text>\n  <text class=\"fig-muted\" x=\"238\" y=\"82\" font-size=\"10.5\">OAI-SearchBot,<\/text>\n  <text class=\"fig-muted\" x=\"238\" y=\"97\" font-size=\"10.5\">PerplexityBot<\/text>\n  <text x=\"238\" y=\"120\" font-size=\"11\">Cost: today&#8217;s citations<\/text><\/p>\n<p>  <rect x=\"434\" y=\"38\" width=\"196\" height=\"94\" rx=\"10\" fill=\"#FFFFFF\" stroke=\"#FF6B35\"\/>\n  <text x=\"450\" y=\"62\" font-size=\"12.5\" font-weight=\"600\">User fetch<\/text>\n  <text class=\"fig-muted\" x=\"450\" y=\"82\" font-size=\"10.5\">ChatGPT-User,<\/text>\n  <text class=\"fig-muted\" x=\"450\" y=\"97\" font-size=\"10.5\">Perplexity-User<\/text>\n  <text x=\"450\" y=\"120\" font-size=\"11\">Cost: the asked-for page<\/text><\/p>\n<p>  <rect x=\"10\" y=\"152\" width=\"620\" height=\"60\" rx=\"10\" fill=\"#F8FAFC\" stroke=\"#E2E8F0\"\/>\n  <text x=\"26\" y=\"176\" font-size=\"11.5\" font-weight=\"600\">A common configuration<\/text>\n  <text class=\"fig-muted\" x=\"26\" y=\"196\" font-size=\"10.5\">Allow search and user fetch. Decide on training separately, as a commercial question.<\/text>\n<\/svg><figcaption>One company can operate agents in all three columns, which is why blocking by company name is a blunt instrument.<\/figcaption><\/figure>\n<h2>The agents, by vendor<\/h2>\n<p>Vendors add, rename and retire user agents, so treat this as a starting point and check the current documentation before committing a policy.<\/p>\n<h3>OpenAI<\/h3>\n<ul>\n<li><code>GPTBot<\/code> \u2014 training collection.<\/li>\n<li><code>OAI-SearchBot<\/code> \u2014 the search index behind ChatGPT&#8217;s live lookups.<\/li>\n<li><code>ChatGPT-User<\/code> \u2014 a fetch triggered by something a user asked.<\/li>\n<\/ul>\n<h3>Anthropic<\/h3>\n<ul>\n<li><code>ClaudeBot<\/code> \u2014 the current crawler.<\/li>\n<li><code>Claude-User<\/code>, <code>Claude-SearchBot<\/code> \u2014 user-initiated and search fetches.<\/li>\n<li><code>anthropic-ai<\/code>, <code>Claude-Web<\/code> \u2014 older names still seen in logs and in copied robots.txt files.<\/li>\n<\/ul>\n<h3>Perplexity<\/h3>\n<ul>\n<li><code>PerplexityBot<\/code> \u2014 indexing for its answer engine.<\/li>\n<li><code>Perplexity-User<\/code> \u2014 a fetch for a specific question.<\/li>\n<\/ul>\n<h3>Worth knowing about<\/h3>\n<ul>\n<li><code>Google-Extended<\/code> \u2014 not a crawler at all. It is a token in robots.txt that controls whether content already crawled by Googlebot may be used for Gemini training. Blocking it does not affect Search.<\/li>\n<li><code>CCBot<\/code> \u2014 Common Crawl. Not an AI company, but its archive feeds many training sets.<\/li>\n<li><code>Applebot-Extended<\/code>, <code>meta-externalagent<\/code> \u2014 the equivalent controls for Apple and Meta.<\/li>\n<\/ul>\n<h2>A worked robots.txt<\/h2>\n<p>This allows the retrieval and user-fetch agents and declines training. It is one defensible position, not the only one.<\/p>\n<pre><code>User-agent: OAI-SearchBot\nAllow: \/\n\nUser-agent: ChatGPT-User\nAllow: \/\n\nUser-agent: PerplexityBot\nAllow: \/\n\nUser-agent: Perplexity-User\nAllow: \/\n\nUser-agent: Claude-SearchBot\nAllow: \/\n\nUser-agent: GPTBot\nDisallow: \/\n\nUser-agent: ClaudeBot\nDisallow: \/\n\nUser-agent: CCBot\nDisallow: \/\n\nUser-agent: Google-Extended\nDisallow: \/<\/code><\/pre>\n<p>Two things to understand before pasting it. Directives are matched per user agent, so a group you do not name falls through to your <code>User-agent: *<\/code> rules. And robots.txt is a request, not a control: named crawlers from large vendors honour it, anonymous scrapers do not, and nothing in a text file protects content that should not be public.<\/p>\n<h2>Checking what is actually reaching you<\/h2>\n<p>Your access log is the only source of truth about who is crawling. On this server:<\/p>\n<pre><code>grep -ioE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|PerplexityBot|CCBot' \\\n  \/var\/log\/nginx\/access.log | sort | uniq -c | sort -rn<\/code><\/pre>\n<p>If a bot you believe you allowed does not appear at all, check robots.txt before concluding anything else. If one you blocked keeps appearing, check whether the user agent string changed.<\/p>\n<div class=\"kb-faq\">\n<h2>Common questions<\/h2>\n<details>\n<summary>Should I block AI crawlers?<\/summary>\n<p>It depends what your content is for. If the content is the product and you sell access to it, blocking training crawlers is defensible. If your content exists to market something else, blocking removes you from the answers people now ask instead of searching, which is usually the more expensive choice.<\/p>\n<\/details>\n<details>\n<summary>Does blocking GPTBot remove me from ChatGPT?<\/summary>\n<p>Not from its live answers. GPTBot is the training collector; live lookups go through the search agent. Blocking GPTBot while allowing OAI-SearchBot keeps you citable today and out of the next training run, which is what most people actually want.<\/p>\n<\/details>\n<details>\n<summary>Does Google-Extended affect my Google rankings?<\/summary>\n<p>No. It governs whether Google may use already-crawled content for generative AI training. Googlebot and Search are unaffected, which is what makes it the least costly of these opt-outs.<\/p>\n<\/details>\n<details>\n<summary>Can I allow some pages and block others?<\/summary>\n<p>Yes, with normal path rules per user agent. A common split is allowing documentation and product pages while disallowing a members area or a large reference dataset you would rather not have absorbed wholesale.<\/p>\n<\/details>\n<details>\n<summary>Do these bots respect crawl-delay?<\/summary>\n<p>Inconsistently, and it is not part of the original robots standard. If a crawler is hitting you hard enough to matter, rate-limit it at the server or CDN rather than asking politely in a text file.<\/p>\n<\/details>\n<\/div>\n<h2>Where to go next<\/h2>\n<p><a href=\"\/kb\/how-does-chatgpt-choose-sources\/\">How does ChatGPT choose sources?<\/a> covers what happens after the fetch succeeds. <a href=\"\/kb\/how-to-add-llms-txt-to-wordpress\/\">How to add llms.txt to WordPress<\/a> is the adjacent convention.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>These are three of the crawlers that read your site on behalf of AI products. They are not interchangeable, and the most common mistake is treating them as one thing to allow&hellip;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5],"tags":[],"class_list":["post-28","post","type-post","status-publish","format-standard","hentry","category-technical"],"_links":{"self":[{"href":"https:\/\/wpcontentwriter.com\/kb\/wp-json\/wp\/v2\/posts\/28","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wpcontentwriter.com\/kb\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wpcontentwriter.com\/kb\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wpcontentwriter.com\/kb\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/wpcontentwriter.com\/kb\/wp-json\/wp\/v2\/comments?post=28"}],"version-history":[{"count":0,"href":"https:\/\/wpcontentwriter.com\/kb\/wp-json\/wp\/v2\/posts\/28\/revisions"}],"wp:attachment":[{"href":"https:\/\/wpcontentwriter.com\/kb\/wp-json\/wp\/v2\/media?parent=28"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wpcontentwriter.com\/kb\/wp-json\/wp\/v2\/categories?post=28"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wpcontentwriter.com\/kb\/wp-json\/wp\/v2\/tags?post=28"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}