If you are asking why is my website not showing up on Google, the cause is almost always one of three things: Google has not indexed the page, something is blocking it, or the page is simply too new. Work through them in that order: confirm the index status in Search Console, rule out robots.txt rules, noindex tags and server-level blocks, then give a new page time.
Why is my website not showing up on Google: is it indexed?
Start here, because every other fix depends on the answer. A page that is not in Google's index cannot appear in results, however good it is.
We begin with a quick site: search for the domain in Google. It is a rough check, not a report, but if nothing comes back the problem is indexing, not ranking.
Then open Search Console. Google's URL Inspection tool help page says the tool “provides information about Google's indexed version of a specific page, and also allows you to test whether a URL might be indexable.” It is the tool Google points to for troubleshooting a missing page, while noting that “There can be many reasons why a page hasn't been indexed.”
Read its verdict carefully. “URL is unknown to Google” means Google has not seen the address before, so the problem is discovery. “URL is on Google” means the page is eligible to appear, but the same help page warns it “doesn't guarantee that your page is appearing in Search results.” Eligible and visible are different things.
For the whole site, use the Page indexing report. It shows how many URLs have been crawled and indexed. Google is explicit that it “isn't used to investigate the index status of specific pages”; for a single page, go back to URL Inspection.
Is something blocking Google from your site?
If the page is known but not indexed, look for a block. There are three kinds, and each one hides in a different place: robots.txt, a noindex rule, and the server itself.
| Block | Where it lives | What Google does | How we check it |
|---|---|---|---|
| robots.txt rule | A text file at the root of the domain | Does not crawl the blocked URLs, though a linked URL can still be indexed without a description | Read the file, then test the URL in URL Inspection |
| noindex rule | A meta tag in the page, or an X-Robots-Tag HTTP header | Drops the page from results once it crawls the page and sees the rule | View the source and the response headers; URL Inspection shows the HTML Googlebot received |
| Server-level block | Firewall, CDN or hosting rules | Cannot reach the page at all | Request the page with a crawler user agent and check the status code; check server logs |
A robots.txt check only tells you what the file says, not what your server does when a crawler arrives.
Does robots.txt stop a website showing on Google?
It stops Google crawling the blocked URLs, which is usually enough to keep your content out of results. Google's robots.txt guide says a robots.txt file “tells search engine crawlers which URLs the crawler can access on your site.”
The same guide makes a point that surprises people: robots.txt “is not a mechanism for keeping a web page out of Google.” A disallowed URL can still be indexed if other sites link to it, and when that happens the result appears without a description.
So when a website is not showing on Google, robots.txt is the first file to read. A single leftover rule from a staging site, disallowing everything, will keep Googlebot away from every page.
Is a noindex tag hiding your pages?
Check for one on every page that is missing. Google's noindex documentation describes noindex as a rule set with either a meta tag or an HTTP response header, and says that when Googlebot extracts it, Google “will drop that page entirely from Google Search results, regardless of whether other sites link to it.”
The header version is easy to miss, because it never appears in the page source. Google calls it the X-Robots-Tag header, and it can apply to non-HTML files such as PDFs as well.
There is a trap when the two blocks overlap. Google says that for noindex to work, the page “must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler.” If robots.txt blocks the page, Google never sees the noindex. To test your implementation, Google recommends the URL Inspection tool, which shows the HTML Googlebot received.
Can a firewall stop a website showing on Google?
Yes. A firewall or CDN rule acts before any page is served, so a crawler it rejects never reaches robots.txt or your content. A robots.txt checker will not see it, because it reads the rules rather than sending a request.
Google's noindex page makes the point in its own terms: if the crawler can't access the page, it never sees what is on it. That is why we always send a real request with a crawler user agent and read the status code. A 200 means the page was served. A 403 means something refused it. A 500 or 503 means the server failed.
What we have seen
On one SEO Autopilot engagement, a US healthcare analytics company on WordPress behind a firewall, the firewall returned 403 to OpenAI's and Anthropic's search crawlers on every page. The trigger was a generic bad-bot rule matching the substring “searchbot”. Eleven other crawlers passed. The client's monitoring tool reported all sixteen AI crawlers as accessible, because it read robots rules instead of sending requests.
Those were AI crawlers, not Googlebot, but the lesson carries over. A rule written to stop bad bots can catch a crawler you want, and a tool that only reads robots.txt will tell you everything is fine. If your pages are missing from Google and robots.txt and noindex are clean, test the server response next.
Is your website just too new for Google?
Sometimes nothing is wrong. Google's SEO Starter Guide says “the vast majority of sites listed in our results are found and added automatically as we crawl the web,” and that Google “primarily finds pages through links from other pages it already crawled.” A brand-new site with no links pointing to it gives Google little to follow.
The same guide is candid about timing: “Some changes might take effect in a few hours, others could take several months.” It also says there is no guarantee that any particular site will be added to Google's index, though sites that follow the Search Essentials are more likely to show up.
To give a new site the best start, we do the following:
- Verify the site in Google Search Console and submit an XML sitemap.
- Link every important page from at least one page Google already crawls, starting with the homepage.
- Use URL Inspection to request indexing for the few pages that matter most. The tool has a daily limit per property, so do not spend it on every URL.
- Earn a handful of genuine links from related sites, so Google has a route in from pages it already visits.
Why is my website not showing up on Google? A step-by-step check
Here is the diagnostic order we use when someone asks why their website is not showing up on Google. Stop at the first step that finds a problem, fix it, and re-check.
- Search for the domain. A site: search tells you, roughly, whether anything is indexed at all.
- Inspect the URL. In Search Console, run URL Inspection on the missing page and note whether it is unknown, not indexed, or on Google.
- Read robots.txt. Look for a disallow rule that covers the page or the whole site.
- Look for noindex. Check the meta tags and the X-Robots-Tag header.
- Test the server. Request the page with a crawler user agent and read the status code. Anything other than 200 needs explaining.
- Check the reason in the Page indexing report. Google says not indexed is not necessarily bad, so read the reason given before you act.
- Give it time. If everything is clean, link the page internally, list it in your sitemap, request indexing, and wait a few weeks before judging.
On the last step, the Page indexing report help says that if you ask Google to validate a fix, validation “typically takes up to about two weeks, but in some cases can take much longer.” It also says not to expect every URL on your site to be indexed: duplicates and thin URLs can be left out for good reasons. Just make sure the key pages are in.
Being found on Google and being cited by ChatGPT, Claude or Perplexity depend on the same kind of access, and a server can treat those crawlers differently from Googlebot. Our free AI visibility checker reads robots.txt rules for 11 AI crawlers and requests your homepage once with each crawler's user agent. That is the checker's own request, not the real crawler's, and it covers the homepage only, so treat it as a first look, not proof of access.
If you would rather have this checked every month, SEO Autopilot is managed SEO and AI search visibility for one site at ₹20,000 a month (about US$230), no setup fee and no minimum term. For related problems, see what “discovered – currently not indexed” means and how a firewall blocked AI crawlers.
Frequently Asked Questions
Why is my website not showing up on Google search?
Usually because the page is not indexed, something is blocking Google, or the site is new. Check the page in Search Console's URL Inspection tool first, then look for robots.txt rules, noindex tags and server errors.
How do I check if Google has indexed my website?
Run a site: search for your domain for a rough answer, then use the URL Inspection tool in Google Search Console for a specific page. For the whole site, the Page indexing report shows how many URLs are crawled and indexed.
How long does it take Google to index a new website?
Google's SEO Starter Guide does not give a fixed time. It says some changes take effect in a few hours and others could take several months, and suggests waiting a few weeks before judging whether your work had an effect.
Does robots.txt remove my site from Google?
Not reliably. Google says robots.txt is not a mechanism for keeping a page out of Google, and a disallowed URL can still be indexed if other sites link to it. To keep a page out, Google recommends noindex or password protection.
Can a firewall stop Google from indexing my site?
It can. A firewall or CDN rule that refuses a crawler acts before any page is served, so robots.txt and noindex never come into play. Request the page with a crawler user agent and check the status code to find out.