The robots.txt rules we ship for AI crawlers (copy them)

Our robots.txt AI template sorts AI crawlers into three groups: search crawlers you allow (OAI-SearchBot, Claude-SearchBot, PerplexityBot), fetchers that act on a user's request (ChatGPT-User, Claude-User, Perplexity-User), and training crawlers you decide on deliberately (GPTBot, ClaudeBot). Every token comes from the vendor's own documentation and the grouping follows RFC 9309. Copy it, change the paths, then check your firewall.

The robots.txt AI template, ready to copy

This is the file we start from. It is plain text, and every line is either a comment, a user-agent line, a rule, or the sitemap location. Swap the placeholder domain and the two example private paths for your own before you publish it.

# robots.txt AI template (Aglocom)
# Replace example.com and the private paths with your own.

# Group A: AI search crawlers. Allow these if you want to be cited.
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /admin/
Disallow: /cart/

# Group B: fetchers that act on a user's request.
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Allow: /
Disallow: /admin/
Disallow: /cart/

# Group C: AI training crawlers. Your decision (see below).
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

# Everyone else, including search engines.
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /cart/

Sitemap: https://www.example.com/sitemap.xml

Three things about it are deliberate. Each vendor's bots are split by job, not by company. The private paths are repeated inside every group rather than written once. And the training group is the only one we expect clients to change, because it is a business decision rather than a technical one.

Which user agents belong in which group?

Put a bot in the search group if its vendor says it powers search answers, in the fetcher group if it only visits a page because a person asked, and in the training group if its vendor says it collects content for model training. The vendors spell this out themselves.

OpenAI's crawler overview says each of its settings is independent, so a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot. That is the single most useful sentence for anyone reading about the GPTBot user agent: blocking it is not the same as leaving ChatGPT search. The same page says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. We go through each token in more detail in our post on what GPTBot does and our AI crawlers list.

Anthropic's help article describes the same three roles: ClaudeBot for training, Claude-SearchBot for search quality, and Claude-User for requests a person makes. It warns that disabling Claude-SearchBot or Claude-User may reduce your visibility in user search results. Perplexity's crawler page says PerplexityBot is not used to crawl content for AI foundation models, which is why it sits in the search group and nowhere else.

This template only covers the three vendors whose crawler pages we link here. If you want to add another bot, take its token from that vendor's own documentation, not from a list someone else compiled.

How do crawlers choose a group in this robots.txt AI template?

A crawler obeys the one group whose user-agent line matches its own token, and ignores the rest. The Robots Exclusion Protocol, RFC 9309, says crawlers must use case-insensitive matching to find that group, and must fall back to the * group only if no matching group exists.

That rule is why the private paths appear three times. A bot named in Group A never reads the * group, so a Disallow: /admin/ written only under User-agent: * does nothing for OAI-SearchBot. This is a mistake we check for whenever we review a robots.txt for AI: a site owner adds a friendly named group for a search bot and, without meaning to, opens every path they had closed for everyone else.

The other grouping rules worth knowing:

• A group is one or more user-agent lines followed by one or more rules, so several bots can share one block, as they do above.
• If more than one group matches the same bot, RFC 9309 says the rules must be combined into one group. Avoid relying on that; one group per bot is easier to read.
• Within a group, the most specific match wins, meaning the rule path with the most octets. Disallow: /admin/ beats Allow: / for anything under /admin/.
• Google's robots.txt specification says that when rules conflict, Google uses the least restrictive rule, and RFC 9309 says an equivalent allow should win over a disallow.

Should you block AI training bots?

Block training crawlers only if you have decided you do not want your public pages used for model training; blocking them does not, by the vendors' own descriptions, remove you from their search answers. OpenAI says disallowing GPTBot indicates your content should not be used in training, and Anthropic says restricting ClaudeBot signals that future materials should be excluded from training datasets.

Our template ships Group C as Disallow: / because it is the safer default to publish before anyone has asked the question. If the business is content to allow training, replace Group C with this:

# Group C, if you are happy for your public pages to be used in training:
User-agent: GPTBot
User-agent: ClaudeBot
Allow: /
Disallow: /admin/
Disallow: /cart/

Either way, keep Group A open if being cited by ChatGPT, Claude or Perplexity matters to you. That is the part of robots.txt for AI that affects visibility; the training group is a separate choice.

How to deploy the robots.txt AI template and test it

Publish it at the root of every host you care about, confirm it returns a clean response, and then test each bot separately. The order we follow:

1. Save the file as /robots.txt, all lowercase, in the top-level path of the site. RFC 9309 says it must be UTF-8 and served as text/plain.
2. Repeat it on every subdomain and protocol you serve. Google says rules apply only to the host, protocol and port where the file is hosted, and Anthropic asks site owners to do this for every subdomain they want to opt out.
3. Keep it under Google's 500 KiB limit. A template like this one is a few hundred bytes.
4. Check the status code. Google treats 4xx errors, except 429, as if no robots.txt existed. RFC 9309 says a server error, such as a status in the 500-599 range, means crawlers must assume complete disallow. Under RFC 9309, a robots.txt that returns 503 tells compliant crawlers to stay out entirely.
5. Allow roughly a day before judging the result. OpenAI says it can take about 24 hours for its search systems to adjust, and Perplexity says changes may take up to 24 hours.
6. Run a request with each user agent against your real pages, not just the homepage, and look for a 200.

For a quick first pass, our free AI visibility checker reads your robots.txt rules for 11 AI crawlers and requests your homepage once with each crawler's user agent. That request comes from the checker, not from the real crawler, and it covers the homepage only, so treat a pass as a good sign rather than proof that OpenAI or Anthropic can reach the site.

Why robots.txt is advisory, and where a firewall overrides it

robots.txt is a request, not a lock: RFC 9309 says the rules are not a form of access authorization and not a substitute for valid content security measures. It also works in the other direction. A file that welcomes a bot means nothing if your firewall refuses it.

Two of the fetchers in Group B are explicit about this. OpenAI says robots.txt rules may not apply to ChatGPT-User because a user initiated the action, and Perplexity says Perplexity-User generally ignores robots.txt rules. If a path must stay private, protect it with authentication, not a Disallow line.

On the firewall side, OpenAI and Perplexity both recommend allowing their published IP ranges alongside robots.txt, and Perplexity says a site behind a WAF may need to explicitly whitelist its bots. Anthropic says blocking its IP addresses may not reliably work as an opt-out, because it stops the bot reading your robots.txt.

What we have seen

On one SEO Autopilot engagement, a US healthcare analytics company on WordPress behind a firewall, the firewall returned 403 to OpenAI's and Anthropic's search crawlers on every page. The trigger was a generic bad-bot rule matching the substring "searchbot". Eleven other crawlers passed. The client's monitoring tool reported all sixteen AI crawlers as accessible, because it read robots rules instead of sending requests. The robots.txt was fine; the firewall was not. We wrote up the pattern in how a firewall blocked AI crawlers.

Frequently Asked Questions