AI crawlers: GPTBot, ClaudeBot and PerplexityBot explained

AI crawlers fall into two families that most robots.txt files treat as one. Training crawlers such as GPTBot, ClaudeBot and CCBot collect content that may shape future models. Answer crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot feed the live answers where your brand can be cited today. Block the second family and you disappear from AI answers, not from AI training.

Two families of crawler, two very different decisions

When marketers say "AI is crawling my site", they are usually describing three different events. A training crawler downloads your pages so their content can join the corpus a future model learns from. A search-index crawler downloads your pages so an assistant can retrieve and cite them in live answers. A user-triggered fetcher grabs a single URL because a person just asked about it in a chat.

These feed different systems on different timescales. Training shapes models that ship months later, so blocking it today changes future models rather than current answers. Indexing and live fetching decide whether your brand can be quoted this afternoon. Every robots.txt decision should therefore start with one question: which of the three am I actually trying to stop? What the assistant then does with the retrieved material is covered in our explainer on query fan-out, and where those quoted sources come from in where AI citations come from.

Blocking a training crawler is a licensing decision. Blocking an answer crawler is a visibility decision.

The main AI user agents and what each one feeds

• GPTBot (OpenAI), training. OpenAI's crawler documentation describes GPTBot as the crawler that collects public web content which may be used to train its foundation models. Disallowing it keeps your future content out of that training set and nothing more.
• OAI-SearchBot (OpenAI), live answers. It builds the search index behind ChatGPT search. Per OpenAI's documentation, it exists so ChatGPT can surface and link to sites in results, and it is not used to collect training data. Block it and your pages stop being citation candidates.
• ChatGPT-User (OpenAI), user actions. It fetches a specific URL when a person asks ChatGPT about it. It behaves like a proxy for a human reader rather than a bulk crawler.
• ClaudeBot (Anthropic), training. Anthropic's documentation lists ClaudeBot as the crawler collecting content that may train and improve future Claude models.
• Claude-SearchBot (Anthropic), live answers. The same documentation describes it as indexing content to improve the quality of the search results Claude cites. Anthropic also documents Claude-User, its equivalent of a user-triggered fetch.
• PerplexityBot (Perplexity), live answers. Perplexity's documentation presents it as the crawler building the index its answer engine cites, not a collector of training data for foundation models.
• Google-Extended (Google), a control token. It has no user-agent string of its own and fetches nothing. Ordinary Googlebot does the crawling, and the Google-Extended token in robots.txt governs whether that content can be used for Gemini training and grounding. Google's documentation states the token does not affect a site's inclusion or ranking in Google Search.
• CCBot (Common Crawl), open training corpus. Common Crawl is a nonprofit whose public dataset is a widely used training source across many AI labs, so a CCBot decision is a broad one, not a single-vendor one.

Controlling each crawler in robots.txt

The most common sensible policy for a commercial site is: opt out of training where you want to, stay wide open to everything that feeds live answers. In robots.txt that looks like this:

# Opt out of model training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Stay visible in live AI answers
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Three caveats. First, robots.txt is a request, not a firewall: the major operators document compliance, but nothing in the protocol enforces it. Second, the Google-Extended block removes your content from Gemini training and grounding while leaving classic Google Search untouched, which is exactly what makes it useful. Third, robots.txt says only what crawlers may fetch, not what your site is about: pairing it with an llms.txt file and clean structured data covers the "what should machines make of this" half of the job.

The trap: blocking everything by default

On 1 July 2025 Cloudflare announced that AI crawlers are blocked by default for every new domain on its network, alongside a pay-per-crawl model for publishers. The rationale was fair: Cloudflare's own June 2025 figures put OpenAI's crawl-to-referral ratio at roughly 1,700 pages crawled per visit sent back, and Anthropic's at around 73,000 to 1. For a publisher monetising page views, that is a bad trade.

For a brand that wants to be recommended by assistants, the same default is a trap. The block does not distinguish between a training crawler scraping your archive and OAI-SearchBot deciding whether you exist as a citable source. It lives in a CDN or WAF dashboard, not in robots.txt, so it never shows up when someone audits your crawl directives. Teams then spend months producing content optimised for Generative Engine Optimization while their edge configuration silently returns 403 to every crawler that could have cited it. If your citations dropped or never appeared, the bot-management console is the first place to look, before any content diagnosis. How assistants pick sources in the first place is the subject of where AI citations come from, and how retrieved pages become quoted answers is covered in grounding in AI search.

How to verify crawler visits in your logs

Your server access logs settle every argument about whether AI crawlers see your site. Filter for the user-agent tokens:

grep -iE "gptbot|oai-searchbot|chatgpt-user|claudebot|claude-searchbot|claude-user|perplexitybot|ccbot" access.log

Then read the results for three things:

1. Presence and frequency. Which agents visit at all, how often, and which sections of the site they favour. An answer crawler that never arrives is a visibility problem in itself.
2. Status codes. Rows of 403 or 429 responses to OAI-SearchBot or Claude-SearchBot are the default-block trap made visible. If you are behind a CDN, check the edge logs too, because a request blocked at the edge never reaches your origin log at all.
3. Authenticity. A user-agent string can be typed by anyone, so verify suspicious traffic against the IP ranges that OpenAI, Anthropic and Perplexity publish for their crawlers before you treat it as genuine, and before you whitelist anything by string alone.

Run the check quarterly and after every CDN or firewall change. It takes minutes and it is the closest thing this discipline has to a ground truth, which is why it is part of every Neurise SEO and GEO audit. For how crawler access fits the wider picture of search, answers and optimisation, see SEO vs AEO vs GEO.

Common questions

Does blocking GPTBot remove my site from ChatGPT answers?

No. GPTBot only feeds training data, while live ChatGPT answers come through OAI-SearchBot and ChatGPT-User. You would need to block all three to disappear entirely, and most brands should block none of the last two.

What is the difference between ClaudeBot and Claude-SearchBot?

According to Anthropic's documentation, ClaudeBot collects content that may be used to train future Claude models, while Claude-SearchBot indexes content to improve the search results Claude cites in live answers.

Should I block CCBot?

Only if you have decided you want your content out of open training corpora. CCBot feeds Common Crawl, a public dataset many AI labs use, so blocking it is a broad opt-out from future models rather than from any live answer.

How do I know which AI crawlers visit my site?

Filter your server or CDN access logs for the user-agent tokens, then validate suspicious hits against the IP ranges the operators publish, because a user-agent string on its own can be spoofed.

Neurise editorial team · published 1 September 2026