How to optimise your website for AI search

If you want to know how to optimize for AI search, start with the work Google and Microsoft say still matters: let the crawlers in, publish pages as crawlable text, answer real questions plainly near the top, and keep any structured data matching what the page shows. There is no secret file or special markup. It is good SEO, done carefully, plus a check that AI crawlers can actually reach you.

How to optimize for AI search starts with SEO

It means making your pages easy for AI systems to find, read and quote. The platforms that publish guidance describe it as an extension of SEO, not a separate discipline.

Google’s guide to optimizing for generative AI features says that from Google Search’s perspective, optimizing for generative AI search is optimizing for the search experience, and thus still SEO. The same guide explains that AI Overviews and AI Mode are rooted in Google’s core Search ranking and quality systems, and rely on a technique called grounding to retrieve relevant, up-to-date pages from the Search index.

Microsoft says something similar. Its advice on optimizing content for inclusion in AI search answers states that traditional SEO fundamentals still matter, with crawlability, metadata, internal linking and backlinks remaining essential, but calls them the starting point. The difference it describes is in what gets ranked: in AI search, ranking still happens, but it is less about ordering whole pages and more about which pieces of content earn a place in the final answer.

Can AI crawlers actually reach your site?

Check this first, because nothing else matters if the answer is no. A crawler that receives a block cannot read your content, however good it is.

The search crawlers are separate from the training crawlers. OpenAI’s crawler overview says OAI-SearchBot is used to surface websites in ChatGPT’s search features, that GPTBot is used to crawl content that may be used in training, and that each setting is independent, so you can allow OAI-SearchBot while disallowing GPTBot. It adds that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, and that it can take about 24 hours after a robots.txt update for its systems to adjust.

Anthropic’s help page on its crawlers draws a similar line. It describes Claude-SearchBot as navigating the web to improve search result quality, and says disabling it prevents Anthropic’s system from indexing your content for search, which may reduce your visibility in user search results. ClaudeBot, by contrast, collects content that could contribute to model training. The page also says blocking Anthropic’s IP addresses may not work correctly as an opt-out, because it impedes the bots from reading your robots.txt file.

Two places can block a crawler: robots.txt and your firewall or CDN. The firewall is the one that catches people out: it can return an error to a bot while robots.txt allows it.

A quick first check is our free AI visibility checker. It reads your robots.txt rules for 11 AI crawlers and requests your homepage once with each crawler’s user agent. That is the checker’s own request, not the real crawler’s, and it covers the homepage only, so treat a pass as a good first sign, not proof that every crawler can reach every page. Our post on firewalls that block AI crawlers covers what to look for when the result is not what you expected.

Is your content in crawlable text that AI systems can read?

It needs to be. Google’s guidance makes eligibility for its AI features depend on the same technical basics as ordinary search.

Google’s AI optimization guide says that to be eligible for its generative AI features, a page must be indexed and eligible to be shown in Google Search with a snippet. Its Search Central post on succeeding in AI search spells out what that includes: Googlebot is not blocked, the page works and returns an HTTP 200 status code, and the page has indexable content. The post adds that meeting the technical requirements covers you for search generally, including AI formats.

Snippet controls matter here too. The same post says nosnippet, data-nosnippet, max-snippet and noindex let you set display preferences, and that more restrictive permissions will limit how your content is featured in AI experiences.

Microsoft adds a point about hidden content: its post warns against hiding important answers in tabs or expandable menus, because AI systems may not render hidden content. In practice we move key answers, pricing and service details out of accordions and into the visible page.

How do you write content AI search can quote?

Write in self-contained passages: a clear question as a heading, a direct answer in the first sentence or two, then the detail. That is the form a system can lift without rewriting.

Microsoft describes why this works. Its post says assistants such as Copilot break content into smaller pieces, a process it calls parsing, and that these modular pieces are what get ranked and assembled into answers. It also says direct questions with clear answers mirror the way people search, and that assistants can often lift these pairs word for word. And it names the page title, description and H1 as important signals AI systems use to interpret purpose and scope.

Google’s advice is about substance rather than format. Its post asks for unique, non-commodity content that visitors will find helpful and satisfying, and notes that people in its AI experiences ask longer and more specific questions, including follow-ups.

So to optimize content for AI, we use a simple checklist on every page we write or rewrite:

• One H1 that states what the page is about, matching the title.
• Headings phrased as the questions people actually type.
• A direct answer in the first sentence under each heading.
• Something specific to you: your own data, process or experience.
• Key answers visible on the page, not inside tabs or accordions.
• Terms defined where they first appear, so a passage still makes sense on its own.

Does structured data help with SEO for AI search?

It helps machines label what they are reading, but it is not a requirement, and it must match the visible page.

Google is direct about this. Its AI optimization guide lists overfocusing on structured data as a mistake, saying structured data is not required for generative AI search and there is no special schema.org markup you need to add, while recommending you keep using it as part of your SEO because it helps with rich result eligibility.

Microsoft’s post is warmer, saying it can label content as a product, review, FAQ or event, turning plain text into structured data that machines can interpret with confidence.

The rule that matters is in Google’s Search Central post: if you use structured data, follow the guidelines, make sure all the content in your markup is also visible on the page, and validate it. FAQ markup for questions that appear nowhere on the page is the most common breach we find.

How to optimize for AI search: the step list

Here is the order we work in, drawn from the guidance above and our own practice. Early steps unblock later ones.

1. Confirm AI search crawlers are allowed in robots.txt: OAI-SearchBot and Claude-SearchBot for search, decided separately from the training crawlers GPTBot and ClaudeBot.
2. Request your pages with those user agents and check your firewall or CDN returns 200, not 403. Test more than the homepage.
3. Confirm your key pages are indexed in Google Search Console and Bing Webmaster Tools.
4. Review snippet controls: nosnippet, max-snippet and noindex settings in templates.
5. Fix templates so each page has exactly one H1, a clear title and an accurate meta description.
6. Rewrite key pages into question headings with direct answers, and move hidden answers into visible text.
7. Add structured data that describes only what is visible, and validate it.
8. Publish new pages that answer the specific, longer questions your buyers ask, with your own evidence.
9. Re-check access after any firewall, CDN or hosting change.

If you would rather not run that list yourself, it is the work SEO Autopilot covers for one site at ₹20,000 a month (about US$230), with no setup fee and no minimum term. For more on the format side, see our guide to ranking in AI Overviews.

What we have seen

On one SEO Autopilot engagement, a US healthcare analytics company on WordPress behind a firewall, the firewall returned 403 to OpenAI’s and Anthropic’s search crawlers on every page. The trigger was a generic bad-bot rule matching the substring “searchbot”. Eleven other crawlers passed. The client’s monitoring tool reported all sixteen AI crawlers as accessible, because it read robots rules instead of sending requests.

On the same site, 21 of 39 pages had no H1 because the post template rendered the title as an H2. One template change took pages with exactly one H1 from 9 of 39 to 30. Between the baseline frozen on 15 July 2026 and the re-measure on 18-19 August 2026, across all 39 sitemap URLs, our AI Visibility score went from 18.35 to 66.35 and crawler access from 45 to 82. Performance fell from 48 to 28, because removing the firewall rule took the CDN edge cache with it. That is why step two says to test real requests, and why every access fix needs a speed check after it.

Frequently Asked Questions