Answer engines are search tools that reply to a question with a written answer linking to the pages behind it, rather than a list of results. Google AI Overviews, AI Mode, ChatGPT search and Perplexity work this way. Each picks sources from pages it can already reach: Google from its Search index, OpenAI and Perplexity through their own crawlers. If that crawler cannot fetch a page, it cannot be cited.
What is an answer engine, and how is it different from a search engine?
An answer engine writes the reply itself and cites pages as supporting links. A classic search engine ranks pages and leaves the reading to you. The pages still matter; they have moved from the answer to the evidence for it.
Google’s guide to optimising for generative AI features names the two labels you will see for this work: “AEO” stands for answer engine optimisation and “GEO” for generative engine optimisation. The same guide says that, from Google Search’s perspective, optimising for generative AI search is optimising for the search experience, and thus still SEO. We agree with the practical point. Answer engines do not replace search fundamentals; they add a layer of questions about access and extraction on top of them.
The useful way to understand AI search engines is not by market share or hype, but by asking two plain questions of each one. Where does it get the pages it cites? And which crawler, if any, has to reach your site for that to happen? The rest of this guide answers both, using each vendor’s own description.
How does AI search work inside Google?
Google’s answer engines, AI Overviews and AI Mode, draw on the Google Search index. Google describes them as rooted in its core ranking and quality systems, so a page has to be indexed before it can be used.
Google’s AI Overviews help page describes them as an AI-generated snapshot with key information and links to dig deeper, shown when its systems determine that generative AI can be especially helpful. It calls AI Overviews a core Google Search feature, like knowledge panels, and says they can and will make mistakes. The AI Mode help page calls AI Mode Google’s most powerful AI search experience, says it divides your question into subtopics and searches for each one simultaneously, and says it relies on Google Search’s deep understanding of web information. In some cases it will provide a set of web links instead, if there is not high enough confidence in an AI response.
How does generative search choose what to show, then? The developer guide describes grounding: relying on Google’s core Search ranking systems to retrieve relevant, up-to-date pages from its Search index. It sets two conditions. A page must be indexed and eligible to be shown in Google Search with a snippet, and a site must be included in Search generative AI features in Search Console.
Just as useful is what the guide says you do not need. It says you don’t need new machine readable files, AI text files, markup or Markdown to appear in Google Search, including its generative AI capabilities. It says Google Search ignores LLMS.txt files, and that structured data isn’t required for generative AI search, though it still helps with rich result eligibility. For Google, then, the work is ordinary indexing and quality work.
How do ChatGPT and Perplexity find pages to cite?
Both publish named crawlers for search and say that allowing them is how a site appears in their search results. Both also run a separate, user-triggered fetcher that their documentation says may not follow robots.txt rules.
OpenAI’s overview of its crawlers says OAI-SearchBot is used to surface websites in search results in ChatGPT’s search features, and that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. GPTBot is a different agent: OpenAI says it crawls content that may be used in training its generative AI foundation models. Each setting is independent, so a site can allow OAI-SearchBot while disallowing GPTBot. ChatGPT-User is the agent used when someone asks ChatGPT a question and it visits a page; OpenAI says it is not used for crawling the web automatically, and that robots.txt rules may not apply because a user initiated the action.
Perplexity’s crawler documentation follows the same pattern. It says PerplexityBot is designed to surface and link websites in search results on Perplexity, and is not used to crawl content for AI foundation models. Perplexity-User supports user actions within Perplexity, and because a user requested the fetch, Perplexity says it generally ignores robots.txt rules. Perplexity also says that if you use a Web Application Firewall, you may need to explicitly whitelist its bots, and recommends combining user-agent matching with IP address verification.
The two pages we link do not describe how either engine ranks or chooses between pages once it has fetched them, so we do not claim to know. What they do make clear is the precondition: no fetch, no citation. We cover OpenAI’s search agent in more depth in our guide to OAI-SearchBot.
Which crawler does each of the answer engines rely on?
Google uses its Search index; ChatGPT search uses OAI-SearchBot; Perplexity uses PerplexityBot. The table sets out what each vendor’s own page says, and what that means for your robots.txt and firewall.
| Answer engine | What the vendor says it relies on | Agent that must reach your pages | What to check |
|---|---|---|---|
| Google AI Overviews and AI Mode | Core Search ranking systems retrieving pages from the Search index (grounding) | No separate AI crawler is named on the guide we link; the page must be indexed | Indexed, eligible for a snippet, and included in Search generative AI features in Search Console |
| ChatGPT search | OAI-SearchBot surfaces websites in ChatGPT’s search features | OAI-SearchBot | Allowed in robots.txt and not refused by your firewall or CDN |
| ChatGPT, when a user asks it to visit a page | ChatGPT-User, initiated by the user | ChatGPT-User | Not blocked at the firewall; robots.txt rules may not apply |
| Perplexity search | PerplexityBot surfaces and links websites in search results | PerplexityBot | Allowed in robots.txt and permitted from Perplexity’s published IP ranges |
| Perplexity, when a user asks it to fetch a page | Perplexity-User, supporting user actions | Perplexity-User | Not blocked at the firewall; generally ignores robots.txt |
| OpenAI model training (not an answer engine) | GPTBot crawls content that may be used in training | GPTBot | A separate choice; disallowing it does not affect ChatGPT search |
Two details from the vendor pages are worth acting on. First, robots.txt changes are not instant: OpenAI says it can take ~24 hours for its search systems to adjust, and Perplexity says it may take up to 24 hours. Second, both OpenAI and Perplexity recommend allowing requests from their published IP ranges, which is a firewall setting, not a robots.txt one. For the full list of AI user agents and what each one does, see our AI crawlers list.
What we have seen when a firewall blocked answer engines
A robots.txt file that allows a crawler is not the same as a site that serves it. We saw the gap on one SEO Autopilot engagement, a US healthcare analytics company on WordPress behind a firewall.
On that site the firewall returned 403 to OpenAI’s and Anthropic’s search crawlers on every page. The trigger was a generic bad-bot rule matching the substring “searchbot”. Eleven other crawlers passed. The client’s monitoring tool reported all sixteen AI crawlers as accessible, because it read robots rules instead of sending requests.
Against a rubric frozen at the 15 July 2026 baseline, and re-measured on 18-19 August 2026 by fetching all 39 sitemap URLs, the site’s AI crawler access score went from 45 to 82 and its AI Visibility score from 18.35 to 66.35. Not everything improved: performance fell from 48 to 28, because removing the firewall rule took the CDN edge cache with it, and AI review signals did not move (18 to 18). Access is a precondition, not a guarantee of being cited.
How do you check your site is open to answer engines?
Test with real requests, not only by reading robots.txt. This is the order we work in:
- Confirm your key pages are indexed and eligible to show with a snippet in Google Search Console, and that the site is included in Search generative AI features there.
- Read your robots.txt and note the rules for OAI-SearchBot and PerplexityBot specifically, not just the wildcard group.
- Decide on GPTBot separately. Blocking training does not have to mean blocking search.
- Review your firewall, CDN and bot-protection rules for anything matching broad strings such as “bot” or “searchbot”.
- Allow the published IP ranges that OpenAI and Perplexity point to, combined with user-agent matching, as Perplexity recommends.
- Request a sample of real pages with each search agent’s user agent and confirm you get 200, not 403 or 429.
- After any change, allow time for it to register, then check again.
For a first pass on steps two and six, the free AI visibility checker reads your robots.txt rules for 11 AI crawlers and requests your homepage once with each crawler’s user agent. That request is the checker’s own, not the real crawler’s, and it covers the homepage only, so treat a pass as a signal, not proof that a crawler can reach your site.
If you would rather have access, indexing and content handled as one programme, our AI SEO services cover search and answer engines together. SEO Autopilot is ₹20,000 a month (about US$230) for one site, with no setup fee and no minimum term.
Frequently Asked Questions
What is an answer engine?
An answer engine is a search tool that replies to a question with a written answer and links to its sources, instead of a ranked list of pages. Google AI Overviews, AI Mode, ChatGPT search and Perplexity are examples.
How does AI search work?
AI search retrieves web pages and then generates an answer from them. Google says its generative AI features use grounding, retrieving pages from its Search index with its core ranking systems. OpenAI and Perplexity use their own search crawlers to reach pages.
Which crawler does ChatGPT search use?
OpenAI says OAI-SearchBot is used to surface websites in ChatGPT’s search features. GPTBot is a separate crawler for training data, and ChatGPT-User visits pages when a user asks it to.
Do I need llms.txt to appear in Google AI Overviews?
No. Google’s guide says you don’t need new machine readable files or AI text files to appear in Google Search, including its generative AI features, and that Google Search ignores LLMS.txt files.
Does blocking GPTBot stop my site appearing in ChatGPT search?
Not according to OpenAI. Its documentation says each setting is independent, so a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot for training.