A robots.txt tester tells you which rules a named crawler would obey for a given URL, so you test one user agent at a time: Googlebot, then each AI crawler you care about. Find the group that matches the token, apply the longest matching rule, and note the verdict. Then check the server separately, because a robots.txt pass does not prove the crawler can actually fetch the page.
What does a robots.txt tester actually check?
It checks one question: for this user agent and this URL, do the rules in the file allow or disallow the fetch? Nothing else. It does not request your pages, and it does not know what your server or firewall will answer.
That narrow scope is by design. RFC 9309, the Robots Exclusion Protocol, states plainly that "these rules are not a form of access authorization" and that the protocol "is not a substitute for valid content security measures." A robots.txt file is a set of requests to well-behaved crawlers. Testing it tells you what you asked for, not what happens.
Google describes the same sequence from the crawler's side: in its robots.txt specification, Google says its crawlers download and parse the file before crawling to learn which parts of the site may be crawled. So the file is read first, and the rules inside it decide which URLs a compliant crawler will even try.
How crawlers pick a group: the rule your robots txt test must follow
A crawler obeys exactly one group of rules, chosen by its user-agent token. Get the group wrong and every verdict after it is wrong too.
RFC 9309 defines a group as one or more user-agent lines followed by one or more rules, ending at the next user-agent line or the end of the file. Crawlers must match the token case-insensitively, and if more than one group matches, the matching groups' rules are combined into one. If no group matches, the crawler falls back to the group with User-agent: *. Google's guide to creating a robots.txt file adds that the asterisk matches all crawlers except the various AdsBot crawlers, which must be named explicitly.
This matters for AI user agents in a specific way. If you write a group for OAI-SearchBot that only disallows a private folder, that crawler stops reading your * group entirely. Any rule you meant to apply to everyone has to be repeated in the named group, or the named crawler never sees it.
User-agent: *
Disallow: /staging/
User-agent: OAI-SearchBot
Disallow: /admin/
# OAI-SearchBot obeys only its own group:
# /admin/ is blocked, /staging/ is NOT.How does longest-match precedence decide between Allow and Disallow?
Inside the chosen group, the most specific matching rule wins, and specificity means length. RFC 9309 says "the most specific match is the match that has the most octets," and if an allow and a disallow rule are equivalent, the allow rule should be used.
Google's specification says the same in its own words: crawlers use the most specific rule based on the length of the rule path, and in case of conflicting rules, including those with wildcards, Google uses the least restrictive rule. The order of lines inside a group does not decide the outcome.
| Rules in the group | URL tested | Verdict | Why |
|---|---|---|---|
Disallow: /blog/ and Allow: /blog/public/ | /blog/public/guide.html | Allowed | The Allow path is longer, so it is more specific |
Disallow: /blog/ and Allow: /blog/public/ | /blog/drafts/post.html | Disallowed | Only the Disallow rule matches |
Allow: /page and Disallow: /page | /page | Allowed | Equivalent rules: the least restrictive wins |
Disallow: (no path) | Any URL | Allowed | Crawlers ignore a disallow rule without a path |
The last row comes from Google's specification, which says crawlers ignore the rule without a path. A blank Disallow: line therefore blocks nothing, which surprises people who expected it to block everything.
How to test robots.txt for Googlebot and each AI user agent
Run the same short procedure for every crawler you care about, and write the verdicts down. This is how we test client files before any change ships.
- Fetch
/robots.txtfrom the exact host, protocol and port you want to test. Google's specification says the rules apply only to the host, protocol and port number where the file is hosted, sowwwand the bare domain each need their own check. - Check the response itself. Google says its crawlers treat all 4xx errors except 429 as if no valid robots.txt file existed, meaning no crawl restrictions. RFC 9309 says that if the file is unreachable because of server errors, such as a status in the 500-599 range, the crawler must assume complete disallow. A broken file and a missing file behave very differently.
- Check the size. Google enforces a robots.txt file size limit of 500 kibibytes.
- List the tokens you need to test:
Googlebot, plus the AI user agents relevant to you, such asOAI-SearchBotandGPTBot. - For each token, find its group using the matching rules above: exact named group first, merged if there are several,
*only if nothing matches. - For each important URL (homepage, a key service page, a blog post, a private path), apply the longest matching rule and record allow or disallow.
- Compare the verdicts with what you intended. Any mismatch is a rule to fix before you test the server.
Treat the tokens as separate decisions. OpenAI's crawler documentation says OAI-SearchBot is used to surface websites in ChatGPT's search features, GPTBot crawls content that may be used to train its foundation models, and each setting is independent of the others. The same page notes it can take about 24 hours from a robots.txt update for its search systems to adjust, and that ChatGPT-User visits are initiated by a user, so robots.txt rules may not apply to them. A tester verdict for one of these tells you nothing about the others.
Why a passing robots.txt tester result does not prove access
A pass means your rules allow the crawler. It does not mean your server lets the crawler in. Firewalls, bot-management rules and CDNs sit in front of the site and can refuse a request that robots.txt welcomes.
They usually decide on the User-Agent header. MDN describes it as a characteristic string that lets servers identify the application making the request; Googlebot's, for example, reads Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html). A security rule that pattern-matches that string can block a crawler without anyone touching robots.txt.
What we have seen
On one SEO Autopilot engagement, a US healthcare analytics company on WordPress behind a firewall, the firewall returned 403 to OpenAI's and Anthropic's search crawlers on every page. The trigger was a generic bad-bot rule matching the substring "searchbot". Eleven other crawlers passed. The client's monitoring tool reported all sixteen AI crawlers as accessible, because it read robots rules instead of sending requests.
That is the gap in one sentence: a robots.txt check would have passed, and the crawlers were still refused. On that engagement, the AI crawler access score went from 45 to 82 between the baseline on 15 July 2026 and the re-measure on 18-19 August 2026. Performance fell from 48 to 28, because removing the firewall rule took the CDN edge cache with it, and median time-to-first-byte went from 0.42s to 2.39s. We wrote up the firewall side in how a firewall blocked AI crawlers.
A robots.txt tester checklist before and after you publish
Use this as the short version. The first half is about the file; the second half is about what the server actually returns.
- The file sits at
/robots.txtin the top-level path, all lowercase, as RFC 9309 requires. - It is UTF-8 plain text and under Google's 500 KiB limit.
- Each AI user agent you name has every rule it needs inside its own group.
- No blank
Disallow:line that someone meant to block everything. - Longest-match verdicts recorded for the homepage, a key page and a private path, per token.
- The server returns 200 for real page requests carrying each crawler's user agent, not 403 or 429.
- Server logs show the real crawlers arriving and getting 200 responses.
For a quick first pass on the server side, our free AI Visibility Checker reads your robots.txt rules for 11 AI crawlers, plus llms.txt and the sitemap, and requests your homepage once with each crawler's user agent. That is the checker's own request, not the real crawler's, and it covers the homepage only, so treat a pass as a first check rather than proof of access. Our robots.txt template for AI crawlers is a starting file if yours needs rewriting.
When should you test robots.txt again?
Retest whenever anything in front of the site changes, not only when the file changes. A new firewall rule, a CDN migration, a security plugin update or a host move can each change what crawlers receive while robots.txt stays identical.
We also retest after any edit to the file, after adding a new AI user agent to it, and after a platform or theme change that might ship its own robots.txt. If you would rather have that monitoring done for you, SEO Autopilot runs these checks as part of a managed engagement at ₹20,000 a month (about US$230), one site, no setup fee and no minimum term.
Frequently Asked Questions
What is a robots.txt tester?
A robots.txt tester applies the rules in your robots.txt file to a given user agent and URL and reports whether that crawler is allowed or disallowed. It reads the file only; it does not request your pages or see what your server returns.
How do I test robots.txt for GPTBot and OAI-SearchBot?
Test each token separately. Find the group that matches the token, combine groups if more than one matches, fall back to the asterisk group only if none does, then apply the longest matching rule to each URL. OpenAI documents the two settings as independent.
Does a blank Disallow line block my whole site?
No. Google says crawlers ignore a disallow rule without a path, so an empty Disallow line blocks nothing. To block everything for a user agent, write Disallow: / with the slash.
Why can a crawler be blocked if robots.txt allows it?
Because robots.txt is not access control. A firewall, bot-management rule or CDN can refuse the request based on the User-Agent header, for example with a 403, even when the file allows the crawler.
What happens if my robots.txt returns an error?
Google treats 4xx errors other than 429 as if no robots.txt existed, so it assumes no restrictions. RFC 9309 says that if the file is unreachable because of server errors, crawlers must assume complete disallow.