OAI-SearchBot, GPTBot and ChatGPT-User: the three OpenAI bots

OAI-SearchBot is the OpenAI crawler that surfaces websites in ChatGPT's search features. It is separate from GPTBot, which collects content that may be used for model training, and from ChatGPT-User, which visits a page when someone asks ChatGPT a question. If OAI-SearchBot is blocked, by robots.txt or by a firewall, your pages will not be shown in ChatGPT search answers.

What is OAI-SearchBot?

OAI-SearchBot is the crawler behind ChatGPT search. If you want your pages to be eligible as sources in ChatGPT's search answers, this is the bot that has to get through.

OpenAI's overview of its crawlers says OAI-SearchBot is used to surface websites in search results in ChatGPT's search features. It also says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. That second sentence is the one to remember: the cost of blocking this crawler is visibility, not just crawl volume.

OpenAI's recommendation is short. To help your site appear in search results, it recommends allowing OAI-SearchBot in your robots.txt file and allowing requests from its published IP ranges. Notice there are two halves to that advice. One is a robots.txt rule. The other is network access, which is where firewalls come in, and where we see things go wrong.

OAI-SearchBot vs GPTBot vs ChatGPT-User: what is the difference?

They do three different jobs, and OpenAI controls them separately. One feeds search, one feeds training, and one acts on behalf of a user in a conversation.

The settings are independent. OpenAI's crawler documentation gives the example of a webmaster allowing OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training. So a training opt-out does not have to cost you ChatGPT search, as long as each OpenAI crawler is handled on purpose. We covered the training side in detail in our GPTBot guide.

When people search for a ChatGPT crawler or an OpenAI crawler, they often mean one of these three without knowing which. For search visibility, the one that matters is OAI-SearchBot.

How do I allow OAI-SearchBot in robots.txt?

Give OAI-SearchBot its own group and allow the paths you want found. An explicit group makes your intent clear and stops a broad wildcard rule from deciding for you.

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

That example allows ChatGPT search and declines training. If you are happy for GPTBot to crawl too, give it an Allow rule instead. Either way, the OAI-SearchBot decision is the one that affects whether you can appear in ChatGPT search answers.

Then wait. For search results, OpenAI says it can take about 24 hours from a robots.txt update for its systems to adjust. Do not judge the change by what ChatGPT shows the same afternoon.

robots.txt, though, only tells a crawler what it may fetch. It does nothing about what your firewall does when the request arrives. That is the failure we want you to check for.

Why would a firewall block OAI-SearchBot when robots.txt allows it?

Because the firewall does not read robots.txt. It applies its own rules to each request, and a rule written to stop bad bots can match a crawler you want in.

This is what we found on one SEO Autopilot engagement, a US healthcare analytics company on WordPress behind a firewall. The firewall returned 403 to OpenAI's and Anthropic's search crawlers on every page. The trigger was a generic bad-bot rule matching the substring "searchbot". Eleven other crawlers passed. The client's monitoring tool reported all sixteen AI crawlers as accessible, because it read robots rules instead of sending requests.

A 403 is not ambiguous. MDN's reference for 403 Forbidden says the status indicates that the server understood the request but refused to process it, and that clients receiving a 403 should expect that repeating the request without modification will fail with the same error. In plain terms, OAI-SearchBot was not having a bad day. It was being turned away, every time, by design.

The lesson is not about one vendor or one rule. Any user-agent match written as a loose substring can catch a legitimate crawler whose name happens to contain it. OAI-SearchBot carries the word in its name. We tell every client to treat a broad pattern like that as a likely block on AI search, and to test for it with real requests. Our full write-up of that engagement is in how a firewall blocked AI crawlers.

How to allowlist OAI-SearchBot at the firewall

Find the rule that blocks it, then make sure an allow decision for OpenAI's crawler runs before any block can. The exact screens differ by provider; the logic below is the procedure we follow.

1. Send a request with the OAI-SearchBot user agent to a few real pages, not only the homepage, and record the status codes. A 403 on a page robots.txt allows points to the edge, not the robots file.
2. Open your firewall or WAF event log and find the rule that fired for those requests. Look for user-agent conditions using broad terms such as "bot", "crawler" or "searchbot".
3. Narrow the offending rule so it no longer matches OpenAI's crawler, or add an exception for it. Where you can, base the exception on OpenAI's published IP ranges, which OpenAI recommends allowing, rather than on the user-agent string alone, since anyone can send any user agent.
4. Check rule order. An allow or skip rule that sits below a block rule may never run.
5. Re-test the same pages and confirm they now return 200. Then check that the change did not quietly alter other edge behaviour, such as caching.

If you use Cloudflare, its custom rules documentation explains the mechanics that matter here. Custom rules can perform actions like Block or Managed Challenge on incoming requests, and you can use the Skip action to skip one or more Cloudflare security features. Crucially, custom rules are evaluated in order, and an action like Block stops the evaluation of later rules, so if an earlier rule blocks a request, a later allowance will not run for it.

Cloudflare also documents an example rule that allows traffic from verified bots while challenging other traffic. That rule uses the cf.client.bot field to determine whether a request came from a known good bot or crawler, and its expression is (ip.src.country in {"US" "MX"} and not cf.client.bot). The page we link names Googlebot and Bingbot as examples and does not mention OpenAI's crawlers, so do not assume OAI-SearchBot is covered by a verified-bot exception. Test it.

One caution from the same engagement. Changing a firewall rule can change more than access. On that site, removing the rule took the CDN edge cache with it, and median time-to-first-byte went from 0.42s to 2.39s. Plan the change with whoever owns performance, not just security.

How do I check whether OAI-SearchBot can reach my site?

Check both layers: read the robots.txt group that applies to OAI-SearchBot, then send requests and look at the status code your server returns. A tool that only reads robots rules can tell you a site is open when the firewall says otherwise.

• robots.txt has an explicit OAI-SearchBot group, or you know which wildcard rules it falls under.
• Requests with the OAI-SearchBot user agent return 200 on the homepage and on a sample of inner pages.
• No firewall rule uses a loose substring that matches "searchbot".
• Any allow or skip rule for OpenAI's crawlers sits above the block rules it is meant to bypass.
• OpenAI's published IP ranges are not blocked by geography or reputation rules.

Our free AI visibility checker reads robots.txt rules for 11 AI crawlers and requests your homepage once with each crawler's user agent. That request comes from the checker, not from OpenAI, and it covers the homepage only. A firewall that verifies OpenAI's IP ranges could treat our request differently from the real crawler's, in either direction. Use it as a first check that surfaces a robots rule or an edge block worth investigating, not as proof that OAI-SearchBot reaches your site.

For every other AI bot and its robots.txt token, see our AI crawlers list. If you want both layers checked across the whole site each month, alongside the content work that earns citations, that is part of SEO Autopilot.

Frequently Asked Questions