Where does ChatGPT get its information?

Where does ChatGPT get its information? From two places: content its underlying models were trained on, and web pages fetched live when it searches or when a user asks it to open a page. OpenAI runs a separate crawler for each job, GPTBot, OAI-SearchBot and ChatGPT-User, and each is controlled independently. Blocking one does not block the others.

Where does ChatGPT get its information, in plain terms?

It gets it from what its models learned in training, and from pages it reads on the live web at the moment it answers. For a site owner, both routes start with a crawler reaching your pages, and OpenAI names a different crawler for each route.

OpenAI's overview of its crawlers describes three user agents. GPTBot crawls content that may be used in training OpenAI's generative AI foundation models. OAI-SearchBot is used to surface websites in search results in ChatGPT's search features. ChatGPT-User is the agent ChatGPT or a CustomGPT may use to visit a web page when a user asks a question.

So when someone asks what sources ChatGPT uses, the honest answer has two halves. Part of what it says reflects material its models were trained on, which you cannot inspect or edit after the fact. Part of it comes from pages retrieved for that conversation, and those are the pages you can influence today: whether the right crawler can reach them, and whether what it finds answers the question clearly.

Which OpenAI crawler does what?

Each of the three has a different job, and OpenAI's page says each setting is independent of the others. The table below sets them out alongside Anthropic's equivalents, because the same pattern appears on both vendors' pages.

The Anthropic help centre article draws the same three-way split: a robot for model development, one to search the web, and one to retrieve content at users' direction. Different names, same structure. If you manage a site, that structure is the thing to learn, because it means "allow AI" or "block AI" is never one switch.

Does ChatGPT use Google or Bing?

The OpenAI crawler page we link does not mention Google or Bing at all. What it documents is OpenAI's own crawlers, so we do not make claims about which search partners sit behind ChatGPT's answers, and we would be wary of anyone who states it as settled fact without a source.

What you can act on is the part OpenAI does document. Its page says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, and it recommends allowing OAI-SearchBot in robots.txt and allowing requests from OpenAI's published IP ranges. Whatever else happens upstream, a site that refuses OpenAI's search crawler has closed the door OpenAI says matters for search visibility.

Google is a separate case with its own documentation. Google's guide to generative AI features in Search says AI Overviews and AI Mode are rooted in its core Search ranking and quality systems, and that they retrieve relevant, up-to-date pages from the Search index, a technique also known as grounding. The same guide says that, from Google Search's perspective, optimising for generative AI search is still SEO. That describes Google's own products, not ChatGPT, but it is a useful reminder: the fundamentals of being crawlable, indexable and clear carry across both.

Why blocking one ChatGPT crawler is not blocking all of them

Because each crawler has its own user agent and its own robots.txt tag, a rule aimed at one can leave the others untouched, and a rule aimed at none of them can catch one by accident. Both happen.

OpenAI's page gives the deliberate version as its own example: a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training. It also means a site that has "blocked ChatGPT" in robots.txt may have blocked training and left search open, or the reverse, depending on which name went into the file.

The ChatGPT-User case needs its own note. OpenAI says it is not used for crawling the web in an automatic fashion, and that because these actions are initiated by a user, robots.txt rules may not apply. A robots.txt line is therefore not a complete answer to the question "can ChatGPT read my page?". Anthropic's page also cautions that blocking the IP addresses its bots operate from may not work correctly or persistently guarantee an opt-out, because it impedes their ability to read your robots.txt file. robots.txt is the control both vendors document.

The accidental version is the one that costs visibility without anyone noticing, and it is the one we have actually seen.

What we have seen: a firewall that blocked only the search crawlers

On one SEO Autopilot engagement, a US healthcare analytics company on WordPress behind a firewall, the firewall returned 403 to OpenAI's and Anthropic's search crawlers on every page. The trigger was a generic bad-bot rule matching the substring "searchbot". OAI-SearchBot and Claude-SearchBot both contain it. Eleven other crawlers passed.

The client's monitoring tool reported all sixteen AI crawlers as accessible, because it read robots rules instead of sending requests. On paper, everything was allowed. In practice, the two crawlers that OpenAI and Anthropic describe as the ones for search were refused, and the training crawlers, whose names do not contain that substring, were not. The site had effectively made the opposite choice to the one in OpenAI's own example, without anyone deciding to.

We have written up how that was found and fixed, including what the fix cost in page speed, in The monitoring said AI crawlers were allowed. The firewall blocked them.

How to check which ChatGPT crawlers can reach your site

Check permission and reality separately: read robots.txt for each crawler by name, then send requests and look at the status code your server returns. Finally, confirm in your logs what the real crawlers received.

1. Open your robots.txt and look for rules naming GPTBot, OAI-SearchBot and ChatGPT-User, and Anthropic's ClaudeBot, Claude-SearchBot and Claude-User. Write down which are allowed and which are not.
2. Decide what you actually want. Training and search are separate choices on both vendors' pages, so make each one on purpose.
3. Search your firewall or WAF rules for broad user-agent matches such as "bot", "crawl" or "searchbot" that could catch a crawler you meant to allow.
4. Request a few of your pages with each crawler's user agent and record the status code. A 403 means the server refused the request.
5. Read your server or firewall logs for the real crawlers' requests and the codes they received. The logs are the evidence; the steps above are simulations.
6. After changing robots.txt, allow time before judging the result. OpenAI's page says it can take about 24 hours for its systems to adjust for search results.

For a first pass, Aglocom's free AI Visibility Checker reads robots.txt rules for 11 AI crawlers and requests your homepage once with each crawler's user agent. That is the checker's own request, not the real crawler's, and it covers the homepage only, so it cannot prove a crawler can reach your site. Treat it as a quick way to spot a refusal worth investigating, then confirm in your logs.

What should you do about where ChatGPT gets its information?

Make sure the crawlers you want can reach the pages you want cited, and make those pages easy to lift an answer from. You cannot control what ChatGPT says, and nobody can promise a citation, but you can remove the reasons it could not use your page.

• Allow OAI-SearchBot and Claude-SearchBot if you want to appear in those assistants' search answers.
• Make a deliberate, written decision about GPTBot and ClaudeBot, the training crawlers.
• Test the firewall as well as robots.txt, since a monitoring report built from robots rules alone can miss a refusal.
• Keep doing the basics: Google's guide says a page must be indexed and eligible to be shown with a snippet to appear in its generative AI features.
• Answer the question in the first lines of each page, so a retrieved passage stands on its own.

One thing not to over-invest in: Google's guide says you do not need new machine-readable files, AI text files or special markup to appear in Google Search, including its generative AI features, and that an llms.txt file will neither harm nor help your visibility there because Google Search ignores it. It adds that maintaining one for other services is fine. Spend your effort on access and clarity first. If you would rather have this checked and maintained every month, that is what our AI SEO services cover.

Frequently Asked Questions