The monitoring said AI crawlers were allowed. The firewall blocked them.

AI crawlers blocked by firewall rules can look perfectly healthy in a robots.txt report. On one client site we audited, a generic bad-bot rule matching the substring "searchbot" returned 403 to OpenAI's and Anthropic's search crawlers on every page, while the client's monitoring tool reported all sixteen AI crawlers as accessible. Permission and the server's real response are separate layers, and you have to test both.

What happened on this site

The site belongs to a US healthcare analytics company. It runs on WordPress, sits behind a firewall, and was one SEO Autopilot engagement. The firewall is the client's own, not ours. It returned 403 to OpenAI's and Anthropic's search crawlers on every page. Eleven other crawlers passed.

The trigger was not a rule written about AI at all. It was a generic bad-bot rule matching the substring "searchbot". OpenAI's crawler overview says OAI-SearchBot is used to surface websites in search results in ChatGPT's search features. Anthropic's help centre says Claude-SearchBot navigates the web to improve search result quality for users. Both names contain the string the rule was looking for, so a rule written to stop bad bots caught the two crawlers that decide whether this site could appear in those search answers.

Be precise about what was not refused. GPTBot, which OpenAI describes as the crawler for content that may be used in training its foundation models, does not have "searchbot" in its name, so this rule had nothing to match. The failure was narrow, which is exactly why it was easy to miss: the rest of the crawler list looked fine.

The consequence is spelled out on the same two pages. OpenAI's overview says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. Anthropic's page says disabling Claude-SearchBot prevents its system from indexing your content for search optimisation, which may reduce your visibility in user search results. A firewall refusal is not a robots.txt opt-out, but the crawler still does not get the page.

Why did the monitoring tool say every AI crawler was allowed?

Because it read robots rules instead of sending requests. The client's monitoring tool reported all sixteen AI crawlers as accessible, and as a statement about robots.txt that was true. It simply was not the whole answer.

robots.txt is a statement of permission. OpenAI's overview describes OAI-SearchBot and GPTBot as robots.txt tags that let webmasters manage how their sites and content work with AI, and Anthropic's help centre says its bots honour industry standard directives in robots.txt. A tool that parses that file can tell you what you have permitted. It cannot tell you what your server does when the request arrives, because a firewall rule that matches on user agent sits in front of the site and does not read robots.txt before it acts.

So the monitoring was answering one question well, "are these crawlers allowed?", and silently skipping the question that mattered, "what happens when they knock?". We now treat any crawler report built only from robots.txt as half a test.

How do AI crawlers get blocked by a firewall when robots.txt allows them?

A firewall rule matches something about the request, such as the user agent, and acts on it before your site answers. If the match is broad, it catches crawlers nobody meant to stop.

Cloudflare's custom rules documentation describes rules that filter incoming requests and perform actions like Block or Managed Challenge. It also says custom rules are evaluated in order, and that if an earlier rule blocks a request, later rules will not run for that request. That detail matters for anyone who tries to fix a problem like this with an exception: an allow rule placed after a broad block never runs for the requests the block has already refused.

The same care applies in the other direction. If you are setting up Cloudflare to block AI bots you genuinely do not want, write the match so it names those crawlers, then check what else it catches. And if an opt-out is the real goal, Anthropic's help centre says blocking the IP addresses its bots operate from may not work correctly or persistently guarantee an opt-out, because doing so impedes its ability to read your robots.txt file. robots.txt is the route that page documents.

When we review a firewall for AI crawler access, this is the checklist we work through:

• User-agent matches on broad substrings such as "bot", "crawl", "spider" or "searchbot".
• Rule order: any allow exception for crawlers must sit before the rules that block.
• Managed Challenge or similar actions applied to all automated traffic rather than to named offenders.
• Country or rate rules with no exception for crawlers. Cloudflare's verified-bots example shows one pattern: a rule that challenges requests from a list of countries but uses the cf.client.bot field to let known good bots through.
• Rules added for one incident and never reviewed since.

What does a 403 Forbidden mean for an AI crawler?

It means your server heard the request and said no. MDN's reference for 403 Forbidden defines it as the server understanding the request but refusing to process it, and notes that authenticating or re-authenticating makes no difference.

MDN adds that clients receiving a 403 should expect a repeat of the same request to fail with the same error. For a crawler, that is not a temporary hiccup to wait out; the refusal is a decision, and it will be made again on the next visit until someone changes the rule. MDN also notes that server owners may send a 404 instead of a 403 when they do not want to acknowledge a resource exists, so a crawler test that only looks for 403 can miss a quieter refusal. We record every status code, not only the ones we expect.

How to check whether AI crawlers are blocked by your firewall

Test both layers: read robots.txt for permission, send requests and record the status code, then confirm in your server or firewall logs what the real crawlers received. The logs are the only part of this that is not a simulation.

The procedure we follow:

1. Read robots.txt and list the rules for every AI crawler you care about, including OAI-SearchBot and Claude-SearchBot as well as GPTBot and ClaudeBot.
2. Request several pages, not just the homepage, with each crawler's user agent string, and write down the status code for each.
3. Open your firewall rule list and search the expressions for substrings that appear in crawler names.
4. Check rule order, so an allow exception is evaluated before any rule that blocks.
5. Where you want a crawler in, follow the vendor's own guidance. OpenAI's overview recommends allowing OAI-SearchBot in robots.txt and allowing requests from its published IP ranges.
6. Read your server or firewall logs for the real crawlers' requests and the codes they received.
7. After any change, re-test and measure time-to-first-byte, because a firewall change can alter caching.

If you want a first pass, Aglocom's free AI Visibility Checker reads robots.txt rules for 11 AI crawlers and requests your homepage once with each crawler's user agent. Be clear about what that is: the checker's own request, sent from Cloudflare, covering the homepage only. It is not the real crawler's request, so it cannot prove a crawler can reach your site. A rule built on verified-bot status, like the cf.client.bot field in Cloudflare's example, judges whether a request came from a known good bot, so a borrowed user agent and the real crawler may not be treated the same way. Your logs remain the proof.

What we have seen after the rule came off

Removing that rule was part of a wider engagement, so the scores below reflect all of the work, not one change. The baseline was frozen on 15 July 2026 and re-measured on 18-19 August 2026 by fetching all 39 sitemap URLs, with the scoring rubric frozen at baseline.

• AI crawler access, which carries 20% of the AI Visibility score, went from 45 to 82.
• AI Visibility overall went from 18.35 to 66.35.
• Schema and structured data went from 5 to 78, and llms.txt from 0 to 85.
• Performance fell from 48 to 28, because removing the firewall rule took the CDN edge cache with it. Median time-to-first-byte went from 0.42s to 2.39s.
• AI review signals did not move: 18 before, 18 after.

The performance line is the one to copy into your own plan. The fix for crawler access had a cost somewhere else, and it only showed up because we re-measured everything against the same frozen rubric. If you change a firewall rule, measure speed before and after, and put caching back deliberately rather than assuming it survives. Unblocking a crawler does not promise a citation; it removes one reason you could not get one. This kind of layered testing is the core of our AI SEO services.

Frequently Asked Questions