The technical SEO checklist (including AI crawlers)

A technical SEO checklist is the short list of things that decide whether search engines and AI answer engines can reach, read and index your pages: crawl access, server responses, sitemaps, canonicals, redirects, page templates and Core Web Vitals. Ours follows Google's own documentation, then adds two checks we learned the hard way: test what your server actually returns to crawlers, and check templates for missing H1s.

What is technical SEO, and why does it need a checklist?

Technical SEO is the work that lets crawlers fetch, understand and index your pages, separate from what the pages say. It needs a checklist because the failures are quiet: nothing looks broken to a human visitor, yet a crawler is turned away or sent to the wrong URL.

Content and links get the attention, but they only count once a page can be crawled and indexed. Google's technical SEO overview puts the baseline plainly: resources and pages Google is meant to crawl should not be blocked by robots.txt rules and should be accessible to an anonymous user. Everything else on this list builds on that sentence.

The difference in 2026 is that Google is no longer the only reader. ChatGPT, Perplexity, Claude and Google AI Overviews depend on their own crawlers fetching your pages. The same technical SEO issues that stop Googlebot can stop them too, and some failures only hit them. So every item below is written for both audiences.

The technical SEO checklist: crawl and index controls

Start with the controls that tell crawlers what they may fetch and what they should index. These are the settings most often misunderstood, because robots.txt and indexing are two different levers.

• robots.txt allows what you want crawled. Google's robots.txt guide describes the file as telling crawlers which URLs they can access, and says it is used mainly to avoid overloading your site. Check it for leftover staging rules and blanket disallows.
• robots.txt is not used to hide pages. The same guide says a disallowed page can still be indexed if other sites link to it, and that the result will show without a description. To keep a page out, use noindex or password protection.
• noindex is only on pages you mean to exclude. We regularly find it left on templates after a redesign.
• Key information is in text, not images. Google's overview says to put key information in text rather than graphics. Answer engines quote text; they cannot lift a passage out of a banner.
• Login walls do not cover public content. If a page should rank, an anonymous visitor must be able to load it.

One caution from the robots.txt guide is worth repeating: the file cannot enforce anything. Respectable crawlers obey it; others might not. That is why we treat robots.txt as a statement of intent, and test access separately.

Does robots.txt tell you whether crawlers can reach your site?

No. robots.txt tells you what you have asked crawlers to do. It says nothing about what your server, firewall or CDN actually returns when a crawler arrives, and that is where we have found the most damaging technical SEO issues.

What we have seen: on one SEO Autopilot engagement, a US healthcare analytics company on WordPress behind a firewall, the firewall returned 403 to OpenAI's and Anthropic's search crawlers on every page. The trigger was a generic bad-bot rule matching the substring "searchbot". Eleven other crawlers passed. The client's monitoring tool reported all sixteen AI crawlers as accessible, because it read robots rules instead of sending requests.

The robots.txt file on that site was fine. The blocking happened a layer below it. So the checklist item is not "robots.txt allows AI crawlers"; it is "the server returns 200 to each crawler you care about". We wrote up the full case in how a firewall blocked AI crawlers.

How we test it:

1. List the crawlers you want: Googlebot, Bingbot, and the search and training crawlers of the AI platforms that matter to you.
2. Request a handful of real URLs (homepage, a service page, a blog post) with each crawler's user agent, and record the HTTP status code.
3. Treat anything other than 200 or a deliberate 301 as a finding: 403, 429 and 503 all deserve a look at firewall and bot-management rules.
4. Check server or CDN logs for the real crawlers' requests, because a test from your own machine is not the crawler's own request.
5. Repeat after any firewall, CDN or security plugin change.

If you want a quick first look, our free AI visibility checker reads robots.txt rules for 11 AI crawlers and requests your homepage once with each crawler's user agent. That is the checker's own request, not the real crawler's, and it covers the homepage only, so treat it as a first signal rather than proof that a crawler can reach the site.

Which sitemap, canonical and redirect checks matter?

A clean sitemap, consistent canonicals and correct redirects tell search engines which URL is the real one. Get them wrong and crawlers waste time on duplicates or index the wrong version.

Google's sitemap overview says a sitemap helps search engines discover URLs but does not guarantee every item will be crawled and indexed. It also notes that a properly linked site of about 500 pages or fewer may not strictly need one, though in most cases a site will benefit. We include one on every client site, because it doubles as the list we audit against.

On canonicals, Google's canonicalization guide is clear that some duplicate content is normal and not a spam violation, and that a canonical preference is a hint, not a rule: Google may choose a different page. It also says Google uses the canonical page as the main source to evaluate content and quality, which is why a misplaced canonical can quietly sink a good page.

On redirects, Google's overview says to use a 301 when a page moves permanently and a 302 when the move is temporary, so Google continues to crawl the original page.

Why should a technical SEO checklist include page templates?

Because one template mistake repeats on every page that uses it. Auditing page by page hides this; auditing by template finds it in minutes and fixes it with one change.

What we have seen: on the same healthcare analytics site, 21 of 39 pages had no H1 because the post template rendered the title as an H2. One template change took pages with exactly one H1 from 9 of 39 to 30.

Template checks we run on every site:

• Exactly one H1 per page, carrying the page's main topic.
• A logical heading order beneath it, so a reader or an answer engine can follow the structure.
• A unique title tag and meta description per page, not a sitewide default.
• Structured data generated from the page's real content, and valid.
• Main content present in the HTML, not only injected after load by scripts.
• Internal links in the body copy, not only in the navigation.

Group your URLs by template (home, service, post, category, product) and check one or two of each. If one is broken, assume all of that type are, then confirm with a crawl in a tool such as Screaming Frog.

How do Core Web Vitals fit into technical SEO?

Core Web Vitals measure the real user experience of loading, responsiveness and visual stability. Google recommends good scores for success with Search, so they belong on the checklist, but they are one item among many rather than the whole job.

Google's Core Web Vitals page sets the thresholds: Largest Contentful Paint within the first 2.5 seconds, Interaction to Next Paint under 200 milliseconds, and Cumulative Layout Shift below 0.1. It points site owners to the Core Web Vitals report in Search Console to see how pages perform.

A warning from our own work: fixes interact. On the healthcare site, removing the firewall rule that blocked AI crawlers took the CDN edge cache with it. Performance fell from 48 to 28 and median time-to-first-byte went from 0.42s to 2.39s, even as AI Visibility rose from 18.35 to 66.35. Access came first, but the speed regression became the next item on the list. Re-measure performance after any security or caching change.

How often should you run this technical SEO checklist?

Run the full checklist at least once, then re-run the high-risk items after every change to hosting, firewall, CDN, theme or plugins. Technical SEO issues tend to arrive with deployments rather than with time.

Our order of work on a new site:

1. Server responses to search and AI crawlers on real URLs.
2. robots.txt and noindex settings.
3. Sitemap contents, canonicals and redirects.
4. Templates: H1s, headings, titles, structured data.
5. Core Web Vitals and server response time.

Access sits first because nothing downstream matters if crawlers are turned away. If you are unsure whether a page problem is about crawling or indexing, our guide to indexability versus crawlability separates the two. And if you would rather have someone run this every month, SEO Autopilot covers one site for ₹20,000 a month (about US$230), with no setup fee and no minimum term.

Frequently Asked Questions