Sign inStart free

Cloudflare SEO: the settings that help crawling, the ones that block it, and the AI bot switches to check

Cloudflare speeds up crawling by Google's own account; the risk sits in Bot Fight Mode, firewall rules, Rocket Loader and the AI crawler policies, read in Cloudflare's docs on 22 October 2026.

Published by AI Knows Us · Updated 21 October 2026

Cloudflare does not hurt SEO on its own. Google's own crawling team says a CDN can let Googlebot crawl a site faster and takes load off your server. The SEO risk sits in a handful of settings: bot protection that challenges crawlers, firewall rules that block them, script tweaks that break rendering, and the AI bot switches that decide whether assistants can read your pages at all. Check those five places and Cloudflare is a net gain for search.

Every fact below was read from Cloudflare's developer documentation and Google Search Central on 22 October 2026. Menu paths are as the docs showed them that day.

What Cloudflare does for search, in Google's words

In its December 2024 "Crawling December" post on CDNs, Google explained three things that matter for SEO:

  • Higher crawl capacity. Google's crawlers allow a higher crawl rate when they detect a site is backed by a CDN, because the server can most likely take more requests at once.
  • Cached pages serve faster. When HTML, images, CSS and JavaScript come from the CDN's cache, your server does less work and pages usually load faster for people.
  • The cold cache. The first time a URL is requested, the CDN has nothing cached, so your origin still serves every URL at least once. If you launch thousands of pages at once, expect a heavy crawl for a few days.

The same post has a section titled "When CDNs are overprotective", and that is the real subject of Cloudflare SEO.

How a crawler gets blocked, and why the type of block matters

Google sorts unwanted CDN blocks into hard blocks and soft blocks. The difference decides how much damage is done.

What the crawler receivesWhat Google says happens
HTTP 503 or 429The preferred way to signal a temporary block. It gives you time to react.
Network timeoutTreated as a hard error. Affected URLs can be removed from the index, and crawl rate can drop because the site looks overloaded.
An error message with HTTP 200A soft error. If Google reads it as a hard error the URL is removed; if not, every page showing the same message can be dropped as a duplicate.
A "are you human" challenge pageThe crawler sees only the challenge, not your page. Google strongly recommends sending a 503 to automated clients instead.

So the goal is simple: real search crawlers get your page, and anything you do block gets a clear status code rather than a challenge dressed up as content.

The five Cloudflare settings to check

1. Bot Fight Mode (free plan)

Bot Fight Mode identifies traffic that matches patterns of known bots and issues challenges that make the client do CPU-heavy work. Cloudflare's docs list two limits worth knowing before you switch it on:

  • It cannot be bypassed or skipped with WAF custom rules or Page Rules. It runs outside the Ruleset Engine, so Skip, Bypass and Allow actions have no effect on it.
  • It may challenge API or mobile app traffic, and JavaScript Detections is switched on automatically and cannot be turned off.

If you need exceptions, Cloudflare points you to Super Bot Fight Mode instead. To see what Bot Fight Mode is doing, open Security, then Analytics, then the Events tab. Challenged requests show "Bot Fight Mode" in the Service field.

2. Super Bot Fight Mode (Pro, Business and Enterprise)

Super Bot Fight Mode adds a setting per bot group: definitely automated, likely automated, and verified bots. Custom rules run before it, so a rule with the Skip action can let named traffic through. Cloudflare defines a verified bot as one that identifies itself honestly (signed requests, a published IP list or reverse DNS) and obeys robots.txt. Search engine crawlers are its first example. Keep the verified bots group on Allow.

3. WAF custom rules, rate limits and challenge pages

A rule written to stop scrapers, a country block or a rate limit can catch Googlebot or Bingbot as well. Before you add one, ask what status code it returns and whether it excludes verified bots. Google's advice is to use the IP ranges search engines publish to identify their crawlers, then remove them from blocking rules or allowlist them.

4. Rocket Loader and other script changes

Rocket Loader defers all JavaScript until after the page renders, which can make the first paint faster. Cloudflare's own docs say that if you see JavaScript or jQuery problems you should disable Rocket Loader and retest. For SEO, the test is whether your main content, links and structured data still appear in the rendered page. If a site depends on JavaScript to show content, test this before and after you enable it.

5. Crawler Hints

Crawler Hints, free on every plan, watches for cache misses as a sign that content has changed and sends those URLs to IndexNow. Google is not on IndexNow's list of participating engines (read on 22 October 2026). That list names Bing, Yandex, Seznam, Naver, Yep, the Internet Archive and Amazonbot. So Crawler Hints helps those engines find changes sooner, and your sitemap and internal links still do that job for Google. Cloudflare also notes that Crawler Hints applies to the whole site, so pages you do not want indexed still need a noindex tag or an X-Robots-Tag header.

The AI bot settings, which now matter as much as Googlebot

Cloudflare now lets every customer set policies for AI crawlers by what they do:

  • Search: crawlers that index content to answer questions about it later.
  • Agent: activity acting in real time for a person, such as chat fetch bots.
  • Training: crawlers that take content to train models, including mixed-purpose crawlers used for both training and search.

Each can be set to block on all pages, block only on pages with ads, or allow. Cloudflare's docs say that from 15 September 2026, new domains get updated defaults: Training and Agent bots are blocked on pages that display ads, while Search stays allowed. If your domain was added after that date and your pages show ads, assistants that fetch pages live for a user may be getting blocked there.

Cloudflare can also manage your robots.txt. When it does, it puts its own block before yours. The example in Cloudflare's docs shows "content signals" (search, ai-input and ai-train) and Disallow lines for crawlers such as GPTBot, ClaudeBot, Google-Extended and Applebot-Extended. The Google-Extended line does not touch your Google rankings. Google's crawler documentation says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." It does control whether Gemini and Vertex AI can use your content.

AI Crawl Control, formerly AI Audit, is the dashboard that shows which AI services visit, whether they follow your robots.txt, and lets you allow or block each one.

A 20-minute Cloudflare SEO check

  1. In Google Search Console, run URL Inspection on your home page and a key product or service page. Look at the rendered screenshot. Your page is good; a blank page, an error or a challenge is a problem.
  2. In Cloudflare, open Security Analytics events and filter for known crawler user agents. Any challenge or block on a search engine is worth fixing first.
  3. Check whether Bot Fight Mode is on. If you are on Pro or above, use Super Bot Fight Mode and keep verified bots on Allow.
  4. Read every WAF custom rule and rate limit. Note what each returns and whether verified bots are excluded.
  5. Open yourdomain.com/robots.txt in a browser and read what is actually served, including any block Cloudflare adds.
  6. Review the AI bot policies and decide, on purpose, which of Search, Agent and Training you allow.
  7. If Rocket Loader is on, compare the rendered HTML in URL Inspection with what a visitor sees.

For redirects, hreflang and title changes deployed through Cloudflare Workers, our guide to edge SEO and the rules Google applies to changes at the CDN covers the method and the cloaking line.

When the crawler that matters is an assistant

Most Cloudflare SEO advice stops at Googlebot. A growing share of buyers now ask ChatGPT, Gemini or Perplexity which business to use, and those answers come from pages the assistants could fetch. A single policy switch in Cloudflare can decide whether that happens, and nothing in Search Console reports it. AI Knows Us asks those assistants the questions your buyers ask and shows whether your site is named and which sources they used, so you can tell when a firewall setting is costing you the answer.

Questions about Cloudflare and SEO

Does Cloudflare hurt SEO?

No, not by itself. Google says a CDN can raise crawl capacity and speed up pages. Problems come from bot protection, firewall rules or script settings that stop crawlers seeing the page.

Does Bot Fight Mode block Googlebot?

Cloudflare's docs do not promise either way. They say Bot Fight Mode cannot be bypassed with custom rules and may challenge API and app traffic. Check Security Analytics and URL Inspection after you enable it. If you need exceptions, use Super Bot Fight Mode.

Does Crawler Hints help with Google?

Crawler Hints sends changes through IndexNow, and Google is not a listed IndexNow participant. It helps Bing, Yandex and the other engines on that list. For Google, keep your sitemap current.

Will Cloudflare's managed robots.txt lower my Google rankings?

No. Its Google-Extended line controls Gemini and Vertex AI use only. Google says that token is not used for Search inclusion or ranking. Make sure Googlebot itself is not disallowed.

Read next

Simon-Kucher pricing strategy, what the firm publishes about its own method and what a Singapore buyer should ask before briefing itThe firm publishes its approach, its scale and four named service lines, and publishes no fee, so the useful work before any call is reading what it claims and deciding which decision you are buying.International pricing strategies for a Singapore company, and the corridor that stops one market undercutting anotherA converted home price is the one answer that is almost always wrong, so set a price per market and keep the gaps narrower than the cost of moving a unit between them.Premium pricing strategy for a Singapore business, and the fence that makes the higher number holdCharging permanently more than the credible alternatives works when the buyer can name what the extra money buys, and the published Singapore writing ladder shows exactly how that is built.Red Antler publishes no price, so here is what its own site does publish and how to cost the same scope from SingaporeRead on 29 October 2026, three pages of redantler.com carry a complete deliverable list, three business stages and not one figure about what any of it costs.Hiring a full service content agency, the five stages a retainer has to cover and the two handoffs where the money leaksMost content retainers buy production and quietly leave strategy and distribution with you, so price the stages rather than the word count.Hiring a branding agency for a Singapore financial services firm, and the three registers to search before you fall in love with a nameIdentity work for a licensed firm starts with a clearance search rather than a mood board, and two of the three registers are free to read in a browser today.

Or browse the whole archive.

See what AI says about you.

The first scan is free, it needs no account, and it asks the questions your buyers ask.

Free. No card. We ask 5 real buyer questions on 2 AI apps.

Already sure you want this? Start free. No card needed, and the free plan stays free. Pay only when you want the work done. Or talk to a person if you have a question first. Or read how we fix what it finds.