Robots.txt 'Disallow: /' - what it blocks, including AI bots

A robots.txt disallow all rule is two lines: User-agent: * followed by Disallow: /. It asks every compliant crawler without its own named group not to fetch any URL on that host. It stops crawling, not indexing: Google can still list a blocked URL without a snippet if other pages link to it. To keep pages out of results, use noindex instead.

What does robots.txt disallow all actually look like?

It is one group with a wildcard user agent and a single slash. That slash is the whole site, because every URL path on the host starts with it.

User-agent: *
Disallow: /

Google's guide to creating a robots.txt file explains the parts. A robots.txt file lives at the root of your site and consists of one or more rules, and each rule blocks or allows access for all or a specific crawler to a specified file path on the domain or subdomain where the file is hosted. Unless you say otherwise, all files are implicitly allowed for crawling, so a disallow rule is how you take that permission away.

Three details catch people out:

• The slash matters. Google's robots.txt specification says crawlers ignore a disallow rule without a path. Disallow: on its own blocks nothing; Disallow: / blocks everything.
• It applies to one host. Google says the rules apply only to the host, protocol and port number where the file is hosted. A disallow all on a staging subdomain does not touch the main site, and the reverse is also true.
• The asterisk is not quite everyone. Google's guide notes that * matches all crawlers except the various AdsBot crawlers, which must be named explicitly.

Does Disallow: / remove a site from Google?

No. It stops Google crawling your pages, which is not the same as removing them from search results. A blocked URL can still appear, usually as a bare link with no description.

Google's robots.txt specification is direct about this: Google can't index the content of pages which are disallowed for crawling, but it may still index the URL and show it in search results without a snippet. The content is hidden from Google; the address is not. If other sites link to a blocked page, Google knows the URL exists and can list it.

In our robots.txt seo work, this is the misunderstanding we meet most. A team adds disallow all to hide a staging site or a private section, then later finds some of those URLs listed in search results without a description. The rule did exactly what it says. It just does not say what they assumed.

Robots.txt disallow all vs noindex: which one do you need?

Use disallow all when you want crawlers to stop fetching pages. Use noindex when you want pages out of search results. They solve different problems, and using both on the same page cancels the second one out.

Google's page on blocking indexing with noindex explains the conflict. For noindex to work, the page must not be blocked by robots.txt and has to be otherwise accessible to the crawler. If the page is blocked, the crawler will never see the noindex rule, and the page can still appear in search results, for example if other pages link to it. Google also says that a response header can be used for non-HTML resources, such as PDFs, video files and image files.

So the order matters when you de-index something. Leave it crawlable, add noindex, wait for Google to recrawl it, and only then consider blocking it. Google's page notes that, depending on the importance of the page, it may take months for Googlebot to revisit, and recommends the URL Inspection tool to see the HTML that Googlebot received.

Does robots.txt disallow all also block AI crawlers?

It blocks every crawler that follows the protocol and has no group of its own in your file, which includes AI crawlers that respect robots.txt. A named group for an AI user agent changes that, in either direction.

The rule comes from RFC 9309, the Robots Exclusion Protocol. Crawlers must match their product token case-insensitively and obey the matching group; if no matching group exists, they must obey the group with a * user-agent line, if present. So a bare disallow all applies to any compliant AI crawler you have not named. Name one, and it reads only its own group:

User-agent: *
Disallow: /

User-agent: OAI-SearchBot
Disallow: /admin/

# OAI-SearchBot ignores the * group.
# It may crawl everything except /admin/.

The reverse warning is the one we raise most often. A site that wants to stay in Google but keep AI crawlers out sometimes writes a named disallow all per AI token. That is a legitimate choice, but it is a whole-site block for that crawler, and a crawler that cannot fetch your pages has nothing of yours to read. Before you copy a list of AI user agents into a disallow all, decide crawler by crawler; our piece on whether you should block AI crawlers walks through that decision.

# Blocks only this token, site-wide
User-agent: GPTBot
Disallow: /

What can go wrong with a robots.txt disallow all rule?

The rule itself is simple. The mistakes are about where it ends up and what people expect it to do.

• It ships to production. A staging file with disallow all gets deployed with the rest of the site, and the live host tells every compliant crawler to stay away.
• The file errors instead. Google says its crawlers treat all 4xx errors except 429 as if no valid robots.txt existed, so a missing file means no restrictions. RFC 9309 says that if the file is unreachable because of server errors, such as a status in the 500-599 range, the crawler must assume complete disallow. A server that throws a 503 on /robots.txt can produce a disallow all you never wrote.
• It is treated as security. RFC 9309 says these rules are not a form of access authorization and are not a substitute for valid content security measures. A disallowed path is still public; the file even advertises it.
• A named group overrides it by accident. Add a group for one crawler to unblock a single folder and that crawler stops reading the * group entirely.

Robots.txt is also only one layer. On one SEO Autopilot engagement, a US healthcare analytics company on WordPress behind a firewall, the firewall returned 403 to OpenAI's and Anthropic's search crawlers on every page. The trigger was a generic bad-bot rule matching the substring "searchbot". Eleven other crawlers passed. The client's monitoring tool reported all sixteen AI crawlers as accessible, because it read robots rules instead of sending requests. A clean robots.txt and a blocked crawler can exist at the same time.

How to remove or replace a disallow all safely

If you find disallow all on a live site that should be crawled, fix it in this order. This is the procedure we follow on client sites.

1. Fetch /robots.txt from the exact host and protocol you serve, including the www and bare-domain versions, and confirm it returns 200 as plain text.
2. Replace Disallow: / with the paths you genuinely want kept from crawlers, or with nothing at all.
3. Check every named group, including AI user agents, for its own disallow all.
4. For pages that must stay out of results, use noindex on crawlable pages rather than a disallow rule.
5. Test the file for each token you care about, then confirm the server returns real pages to those user agents, not 403 or 429.
6. Use the URL Inspection tool in Google Search Console on a few key URLs to see what Googlebot received.

Our guide on how to test robots.txt covers step five in detail. For a quick first pass, the free AI Visibility Checker reads your robots.txt rules for 11 AI crawlers and requests your homepage once with each crawler's user agent. That is the checker's own request, not the real crawler's, and it covers the homepage only, so treat a pass as a first check rather than proof of access. If you would rather have this watched for you, SEO Autopilot is ₹20,000 a month (about US$230), one site, no setup fee and no minimum term.

Frequently Asked Questions