Crawlability is whether a search engine can fetch a page; indexability is whether, once fetched, the page is allowed and fit to be stored in the index and shown in results. Robots.txt controls crawling. A noindex rule controls indexing. A 403 or 5xx response from your server stops both in practice, even when robots.txt allows everything. Fix them in that order: access first, then indexability.
What is the difference between crawlability and indexability?
Crawlability is about access; indexability is about permission and eligibility to be stored. They are two separate stages, and a page has to clear the first before the second even comes into play.
Google describes its own pipeline in three stages in its guide to how Google Search works. Crawling is when "Google downloads text, images, and videos from pages it found on the internet with automated programs called crawlers." Indexing is when "Google analyzes the text, images, and video files on the page, and stores the information in the Google index, which is a large database." Serving is the third stage, when Google returns results for a query.
Crawlability covers whether Googlebot can request a URL and get the real content back: robots.txt rules, server responses, firewalls and rendering. Indexability covers what happens after a successful fetch: a noindex rule, duplication, and whether the page is judged worth keeping. During indexing, the same guide says, Google also "determines if a page is a duplicate of another page on the internet or canonical".
One more line from that guide is worth keeping in mind before you fix anything: "Google doesn't guarantee that it will crawl, index, or serve your page, even if your page follows the Google Search Essentials." Fixing both removes obstacles; it does not buy a ranking.
Robots.txt controls crawling, noindex controls indexability
Use robots.txt to say which URLs a crawler may request, and use noindex to say which pages may not be stored in the index. Mixing them up causes many of the problems we are asked to fix.
Google's robots.txt introduction is direct about what the file is for. It "tells search engine crawlers which URLs the crawler can access on your site" and "is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google." That first purpose, managing how much crawling your server absorbs, is the territory people usually call crawl budget. The second sentence is the one teams miss.
For keeping a page out of results, Google points to noindex. Its page on blocking indexing says noindex "is a rule set with either a <meta> tag or HTTP response header", and that when Googlebot extracts it, "Google will drop that page entirely from Google Search results, regardless of whether other sites link to it." The header form, X-Robots-Tag, also works for non-HTML resources such as PDFs, video files and image files.
| Mechanism | What it controls | Stops crawling? | Stops indexing? |
|---|---|---|---|
| robots.txt Disallow | Which URLs a crawler may request | Yes, for crawlers that obey it | No. The URL can still be indexed if linked from elsewhere, without a description |
| noindex meta tag or X-Robots-Tag header | Whether a fetched page may be stored and shown | No. The page must be crawlable for the rule to be seen | Yes, once Googlebot crawls the page and reads it |
| Password protection | Who can see the content at all | Yes, the content is not served | Yes, per Google's robots.txt guide |
| 403 or other 4xx (except 429) | The server refuses or reports nothing there | The content is not delivered | Yes. Google does not index 4xx URLs and removes indexed ones |
| 5xx or 429 | The server fails or asks the crawler to slow down | Crawling slows down | Not at first. Indexed URLs are kept, but eventually dropped |
Why can a page blocked in robots.txt still appear in Google?
Because blocking the crawl does not block the URL from being known. If other pages link to it, Google can index the address without ever reading the content.
Google's robots.txt guide says it plainly: "If your web page is blocked with a robots.txt file, its URL can still appear in search results, but the search result won't have a description." It also notes that Google "might still find and index a disallowed URL if it is linked from other places on the web."
The trap gets worse when a team adds noindex and a robots.txt Disallow to the same page, thinking two locks are better than one. Google's noindex documentation explains why this backfires: "For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler." If the crawler cannot fetch the page, "the crawler will never see the noindex rule, and the page can still appear in search results."
So: to keep a page out of the index, leave it crawlable and add noindex. Robots.txt is also voluntary; Google's guide says "it's up to the crawler to obey them." If something must not be seen, password-protect it.
What do 403 and 5xx errors do to crawling and indexability?
A 403 stops indexing outright, and a run of 5xx errors slows crawling and, if it persists, loses indexed pages too. Either way, the server response overrides whatever robots.txt says.
Google's page on how HTTP status codes affect its crawlers sets out the behaviour. "Google doesn't use the content from URLs that return 4xx status codes." For Google Search, it "doesn't index URLs that return a 4xx status code, and URLs that are already indexed and return a 4xx status code are removed from the index." It adds that all 4xx errors except 429 are treated the same: the crawler reports that the content does not exist.
Server errors behave differently. The same page says "5xx and 429 server errors prompt Google's crawlers to temporarily slow down with crawling", and that already indexed URLs "are preserved in the index, but eventually dropped." That makes 5xx the crawl budget problem in practice: an unstable server gets crawled less, and pages that keep failing fall out over time.
Two more points: a 2xx success "doesn't guarantee indexing", and 4xx codes other than 429 "have no effect on crawl rate", so a 403 used to slow a crawler simply tells it the page is gone.
What we have seen: a firewall block outside robots.txt
The clearest example we have of a server-level block sitting outside robots.txt comes from one SEO Autopilot engagement with a US healthcare analytics company on WordPress behind a firewall.
On that site the firewall returned 403 to OpenAI's and Anthropic's search crawlers on every page. The trigger was a generic bad-bot rule matching the substring "searchbot". Eleven other crawlers passed. The client's monitoring tool reported all sixteen AI crawlers as accessible, because it read robots rules instead of sending requests.
Robots.txt said yes; the server said no, and the server wins. These were AI search crawlers rather than Googlebot, but the mechanism is identical: a crawlability failure that no robots.txt reader can see. If the same class of rule had matched Googlebot, Google's status code documentation is clear that 4xx URLs are not indexed. We wrote up the AI crawler side in detail in how a firewall blocked AI crawlers.
The fix had a cost worth stating. On re-measurement, AI crawler access went from 45 to 82, but Performance fell from 48 to 28, because removing the firewall rule took the CDN edge cache with it (median time-to-first-byte 0.42s to 2.39s). Check what else a rule does before you remove it.
How do you check crawlability and indexability on your own site?
Test access by sending real requests, then test indexability by looking at what Google received. Reading configuration files alone will not catch a firewall or a template fault.
- Open your robots.txt and confirm that no important section is disallowed. Remember this only tells you what you have asked crawlers to do, not what your server does.
- Request key URLs and record the status code each one returns. Anything other than a 2xx on a page you want indexed is a crawlability problem first.
- Check pages you want indexed for a noindex meta tag or an X-Robots-Tag header, including headers set by a plugin, CDN or server config.
- Check pages you want removed: they need noindex and must not be disallowed in robots.txt, or Google will never see the rule.
- In Google Search Console, use the URL Inspection tool. Google's noindex page recommends it "to see the HTML that Googlebot received while crawling the page."
- Watch for patterns of 5xx and 429 errors in Search Console, which signal that crawling is being slowed.
For the AI crawler side, our free AI visibility checker reads robots.txt rules for 11 AI crawlers and requests the homepage once with each crawler's user agent. That is the checker's own request, not the real crawler's, and it covers the homepage only, so treat a pass as a good sign rather than proof that each crawler can reach your site.
If pages are reachable but still missing, the indexability side is where to look next; our posts on Discovered, currently not indexed and why a website is not showing on Google cover the common causes.
Frequently Asked Questions
Is crawlability the same as indexability?
No. Crawlability is whether a search engine can fetch a page. Indexability is whether a fetched page may be stored in the index and shown in results. A page can be crawlable but not indexable, for example if it carries a noindex rule.
Does robots.txt stop a page being indexed?
Not reliably. Google says a URL blocked in robots.txt can still appear in results, without a description, if it is linked from elsewhere. To keep a page out of Google, use noindex or password-protect it.
Should I use noindex and robots.txt on the same page?
No. For noindex to work, the page must not be blocked by robots.txt. If Googlebot cannot crawl the page, it never sees the noindex rule, and the page can still appear in results.
Does a 403 error remove a page from Google?
Yes, per Google's documentation. Google does not index URLs that return a 4xx status code, and already indexed URLs that start returning one are removed from the index. A firewall 403 is treated like any other 4xx.
How do 5xx errors affect crawl budget?
Google says 5xx and 429 errors make its crawlers temporarily slow down. Already indexed URLs are kept at first but eventually dropped if the errors continue, so a failing server costs both crawling and indexed pages.