Cloudflare Bot Fight Mode and AI crawlers: is it blocking them?

Cloudflare Bot Fight Mode can challenge automated traffic it matches to known bot patterns, and it cannot be bypassed with WAF custom rules. But in the one blocked site we have measured, the culprit was not Bot Fight Mode: it was a broad custom rule. Check all three layers, Bot Fight Mode, Block AI Bots and your own WAF rules, before you blame any one of them.

What does Cloudflare Bot Fight Mode actually do?

Bot Fight Mode is Cloudflare's free, switch-on bot filter. In Cloudflare's own description, it identifies traffic matching patterns of known bots and issues computationally expensive challenges that force the requesting client to perform CPU-intensive calculations.

Three details on that page matter for anyone who wants AI search tools to reach their site:

• It cannot be customised. Cloudflare says it cannot be customised, adjusted or reconfigured via WAF custom rules, and that Skip, Bypass and Allow actions have no effect on it, because it does not run on the Ruleset Engine.
• It can catch traffic you wanted. The same page warns it may challenge API or mobile app traffic.
• Exceptions need a different product. For exceptions, Cloudflare points to Super Bot Fight Mode, which runs on the Ruleset Engine and supports Skip rules, or Bot Management for Enterprise for more granular control.

So Bot Fight Mode is an all-or-nothing switch. If you suspect it is challenging a crawler you care about, there is no allow rule you can write beside it; you either turn it off or move to a product that accepts exceptions.

Can Cloudflare Bot Fight Mode block AI search crawlers?

It can challenge any client it matches to a known bot pattern, but the page we link does not list which AI crawlers it matches. That means you cannot settle the question by reading settings. You settle it by sending requests and reading what comes back.

This matters because a challenge is not the same as a polite refusal. A crawler that receives a challenge or a 403 instead of the page has nothing to index. From the site owner's side nothing looks broken: browsers still load the page, the dashboard looks calm, and the only symptom is that AI answers stop drawing on your content.

Our practice is to treat Bot Fight Mode as one suspect among three, not the default explanation. The other two are Cloudflare's AI bot controls and your own WAF custom rules, and in the case we describe below it was the third.

How do Block AI Bots and AI Crawl Control differ?

Block AI Bots is a setting that blocks AI crawlers by what they do; AI Crawl Control is a dashboard for seeing and managing them crawler by crawler. Cloudflare's Block AI Bots page sorts AI traffic into three behaviours: Search (crawlers that collect or index content to answer questions about it later), Agent (automated activity acting in real time on a person's behalf) and Training (crawlers taking content to train or fine-tune a model).

Two lines on that page are easy to miss. First, mixed-purpose crawlers that combine Search and Training are blocked by all configurations that block AI training, including the legacy "Block AI bots" option. Second, from September 15, 2026, Cloudflare sets updated defaults for new domains: Training and Agent bots blocked on pages that display ads, with Search still allowed. If your domain was added recently, check what was switched on for you.

AI Crawl Control, formerly AI Audit, is available on all plans. Cloudflare describes it as showing which AI services access your content, letting you set allow or block rules for individual crawlers, and tracking which crawlers follow your robots.txt directives.

How a broad custom WAF rule blocked AI search bots

A WAF custom rule does exactly what its expression says, to every request that matches. Cloudflare's custom rules documentation describes them as rules that perform actions like Block or Managed Challenge on incoming requests, evaluated in order, where an action like Block stops evaluation, so later rules do not run for that request.

That ordering is why one loose rule can undo careful settings elsewhere. An allow-list for AI crawlers written lower down never gets a chance if a block rule above it has already matched.

What we have seen

On one SEO Autopilot engagement, a US healthcare analytics company on WordPress behind a firewall, the firewall returned 403 to OpenAI's and Anthropic's search crawlers on every page. The trigger was a generic bad-bot rule matching the substring "searchbot". Eleven other crawlers passed. The client's monitoring tool reported all sixteen AI crawlers as accessible, because it read robots rules instead of sending requests.

Fixing access mattered: AI crawler access on that site went from 45 to 82 between the baseline on 15 July 2026 and the re-measurement on 18-19 August 2026. It also had a cost. Performance fell from 48 to 28, because removing the firewall rule took the CDN edge cache with it, and median time-to-first-byte went from 0.42s to 2.39s. We cover the full engagement in how a firewall blocked AI crawlers.

The lesson is narrow and practical: a rule written to catch bad bots by a word in the user agent will also catch any good bot whose name contains that word. Search for substring matches on user agent first.

How do I check whether Cloudflare is blocking AI crawlers?

Send requests, read status codes, then find the rule. Robots.txt tells you what you have asked crawlers to do; only a request tells you what your edge actually does. This is the order we work in:

1. Read your robots.txt and note which AI crawlers you allow or disallow. This is your stated intent, nothing more.
2. Request your pages with each crawler's user agent and record the status code. A 200 is a page; a 403 or a challenge page is a block. Test several templates, not only the homepage, because rules can be path-specific.
3. Open AI Crawl Control and look at crawler activity and the per-crawler policy for the bots you care about.
4. Check Block AI Bots and note which behaviours are blocked, remembering that a training block also catches mixed-purpose crawlers.
5. Check whether Bot Fight Mode is on. If it is and you need exceptions, plan a move to Super Bot Fight Mode rather than writing rules that will have no effect.
6. List every WAF custom rule that matches on user agent, in evaluation order, and look for broad substrings such as "bot", "crawler" or "searchbot".
7. Re-test after each change, and watch performance, since a firewall change can affect caching.

For step 2, our free AI visibility checker reads robots.txt rules for 11 AI crawlers and requests your homepage once with each crawler's user agent. That is the checker's own request from Cloudflare, not the real crawler's, and it covers the homepage only. Treat a pass as a good sign, not proof that a crawler can reach your site.

How do you allow verified bots without opening the door?

Allow by identity, not by name. A user-agent string is just text any client can send. Cloudflare's verified bots page defines a verified bot as one Cloudflare has confirmed is transparent about who it is, declaring its identity through a cryptographic Web Bot Auth signature, a published IP list with a stable user agent, or reverse DNS, and behaving non-abusively, including obeying robots.txt.

The same page notes that, under a taxonomy introduced on July 1, 2026, AI search and traditional search are both treated as Search behaviour, and that historically verified bots were excluded in default bot configurations; customers can now configure AI bot policies to define what they block or allow. A service that breaches the requirements is removed from the global allowlist.

Our checklist for a safe allow, before changing anything else:

• Decide which behaviours you want: Search, Agent, Training. If you want to be cited in AI answers, Search is the behaviour to keep allowed.
• Set that decision once, in Block AI Bots or AI Crawl Control, not in scattered custom rules.
• Replace user-agent substring blocks with rules that exempt verified bots, then block the rest.
• Place any Skip rule above the block rules it needs to override, since evaluation stops at a Block.
• If Bot Fight Mode is on and you need exceptions, use a product that supports Skip rules.
• Re-test with real requests, and record the result so the next rule change can be compared.

If you would rather not run this audit yourself, SEO Autopilot is our managed SEO and AI search visibility service, at ₹20,000 a month (about US$230) for one site, with no setup fee and no minimum term.

Frequently Asked Questions