Should you block AI crawlers? A decision guide

You can block AI crawlers that collect training data and still be cited by AI search, because the major vendors run training and search under separate user agents. Decide on training bots such as GPTBot, ClaudeBot and Google-Extended on their own merits, keep the search and retrieval bots open if you want citations, and check that your CDN or firewall is not blocking more than your robots.txt says.

Should you block AI crawlers at all?

It depends on what you want from AI tools. If you want your pages quoted and linked in ChatGPT search, Claude or Google, the search and retrieval crawlers need access. If you do not want your content used to train future models, the training crawlers are the ones to block, and blocking them is a separate decision.

The mistake we see is treating "AI crawlers" as one switch. A single rule that turns every AI bot off removes you from the answers you might want to appear in, and a single rule that turns them all on hands over training data you may have wanted to keep. The vendors themselves split these jobs, so your robots.txt for AI can split them too.

This is an opinion piece, so here is ours plainly: for a business that sells a service and wants to be found, we recommend keeping search and retrieval bots open and making the training decision deliberately, with the people who own the content. Neither choice is wrong. Making it by accident is.

Which AI crawlers train models and which power search?

Each vendor names its crawlers by purpose. OpenAI's crawler overview says GPTBot is used to crawl content that may be used in training its generative AI foundation models, while OAI-SearchBot is used to surface websites in search results in ChatGPT's search features. Anthropic's help page describes ClaudeBot as collecting web content that could potentially contribute to training, Claude-SearchBot as improving search result quality, and Claude-User as the agent that may access a site when someone asks Claude a question.

Google works differently. Google's common crawlers page says Google-Extended does not have a separate HTTP request user agent string; the token is used in a control capacity in robots.txt. The same page says Google-Extended does not impact a site's inclusion in Google Search, nor is it used as a ranking signal there.

How do you block GPTBot without losing ChatGPT search?

Give GPTBot its own group in robots.txt with a disallow rule, and give OAI-SearchBot a group that allows your public pages. OpenAI's page says each setting is independent: a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training. It also says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.

A minimal version of that split looks like this:

# Training crawlers: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

# AI search and retrieval: allowed
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: ChatGPT-User
Allow: /

# Everyone else
User-agent: *
Allow: /

Two details matter. OpenAI's page notes it can take ~24 hours from a robots.txt update for its search systems to adjust, so do not judge the change the same morning. It also says that because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply to them. The grouping itself follows RFC 9309, the Robots Exclusion Protocol: crawlers must match the group for their product token case-insensitively, and if no group matches they obey the "*" group.

If you want the full annotated file, our robots.txt AI template groups every token we work with and explains each line.

How do you block AI crawlers from Anthropic and Google?

Use the same pattern: name the training token, disallow it, and leave the search and user tokens alone. Anthropic says its bots honour industry standard robots.txt directives, and asks you to add the rules for every subdomain you wish to opt out from.

Anthropic's page is explicit about the cost of going further. It says disabling Claude-SearchBot may reduce your site's visibility and accuracy in user search results, and that disabling Claude-User may reduce visibility for user-directed web search. It also warns that blocking the IP addresses its bots operate from may not work correctly or persistently guarantee an opt-out, because that impedes its ability to read your robots.txt. In practice, robots.txt is the documented route; IP blocks are not.

For Google, Google-Extended is the training-related token in our default above. Because the page we link says it has no separate user agent and does not affect inclusion in Google Search, a Google-Extended rule is not a way to remove yourself from Google results, and blocking Googlebot to make an AI point would cost you search traffic. Google's page also cautions that the user agent string can be spoofed, which is a reason not to trust a log line on its own.

Why can a one-click CDN block catch search bots too?

Because CDN presets group bots by behaviour, and some crawlers do more than one job. Cloudflare's Block AI Bots documentation sorts bots into Search, Agent and Training, and says mixed-purpose crawlers that combine Search and Training will also be blocked by all configurations to block AI training, including the legacy "Block AI bots" option. Its page also says that on September 15, 2026, new domains get updated defaults: bots classified as Training or as Agent are blocked on pages that display ads, while Search remains allowed.

So a CDN toggle can block a crawler that also feeds search answers, and it can block agent-style fetches that your robots.txt explicitly allows. Your robots.txt is then telling one story and your edge is enforcing another. RFC 9309 itself says robots.txt rules are not a form of access authorization; a firewall rule is.

What we have seen. On one SEO Autopilot engagement, a US healthcare analytics company on WordPress behind a firewall, the firewall returned 403 to OpenAI's and Anthropic's search crawlers on every page. The trigger was a generic bad-bot rule matching the substring "searchbot". Eleven other crawlers passed. The client's monitoring tool reported all sixteen AI crawlers as accessible, because it read robots rules instead of sending requests.

We cover the Cloudflare side in more depth in Cloudflare Bot Fight Mode and AI crawlers.

A decision checklist for your robots.txt for AI

Work through these in order. The first three are business decisions; the rest are checks.

• Do you want to be cited in AI answers? If yes, OAI-SearchBot, Claude-SearchBot and the user-initiated agents stay open.
• Do you object to training use? If yes, disallow GPTBot, ClaudeBot and Google-Extended. If you are unsure, decide with whoever owns the content rather than leaving it to a default.
• Is any content off-limits to every bot? robots.txt is not a security measure, says RFC 9309. Put private material behind authentication, not a disallow line.
• Does your CDN or firewall agree with robots.txt? Read every bot, AI and bad-bot rule at the edge and compare it, line by line, with your file.
• Is robots.txt itself reachable? RFC 9309 says that if the file is unreachable due to server errors, crawlers must assume complete disallow. A 500 or 503 on robots.txt blocks everything.
• Did you test with requests, not just rules? Reading rules tells you intent; sending a request tells you what the server does.

Then test it, in this order:

1. Fetch /robots.txt yourself and confirm it returns 200 as plain text.
2. Request a few key pages with each search crawler's user agent and record the status codes. A 403 or 429 where you expected 200 means an edge rule is involved.
3. Check your CDN or firewall event log for blocked requests from those agents.
4. Wait at least a day after a robots.txt change before checking how ChatGPT search treats you.

Our free AI visibility checker is a starting point for the first two steps. It reads your robots.txt rules for 11 AI crawlers and requests your homepage once with each crawler's user agent. That request comes from the checker, not from the real crawler, and it covers the homepage only, so a pass does not prove a crawler can reach the site. It does show when your rules and your server disagree.

If you would rather have this monitored for you, SEO Autopilot is ₹20,000 a month (about US$230) for one site, with no setup fee and no minimum term.

Frequently Asked Questions