Why blocking AI crawlers costs you recommendations

If an answer engine cannot fetch your pages, it cannot name you. Here is how to decide, change and check it, agent by agent.

Published by AI Knows Us · Updated 9 October 2026

The answer, first

Mostly, no: do not block the retrieval crawlers. AI assistants name businesses they can read. If you disallow the agents that fetch pages to answer a live question, you remove yourself from the shortlist those assistants build. You can still refuse training use. Training crawlers and retrieval crawlers are separate user agents at OpenAI, Google, Perplexity and Anthropic, so you can allow one and refuse the other. This page is published by AI Knows Us, which measures how often AI apps name a business.

The problem you are actually solving

Someone on your team read a story about AI scraping. A legal reviewer asked why your words are training a model for free. A quick fix landed in robots.txt or in a firewall rule, and now a blanket block sits on your site. Months later, a founder types your category into ChatGPT, three competitors come back by name, and you are not there.

Both instincts are reasonable. Nobody wants their archive swallowed for free. But a single blanket block treats two very different things as one. Training is a model reading your pages to learn from them, months before anyone asks a question. Retrieval is an assistant fetching your page right now, because a user asked something your page answers, and then citing you. Blocking training costs you nothing in recommendations. Blocking retrieval costs you the recommendation itself.

So the real question is not "should I block AI". It is "which agents, for which purpose, and how do I check what I did".

Why this step and not another

People reach for three other fixes first. Here is why they do less than you hope.

Writing more content

More pages do nothing if the fetcher gets a 403. Fix access before you fix supply. Access is one file and one firewall screen. Content is months.

Adding schema markup

Structured data helps a parser read a page it already has. It cannot help a parser that was turned away at the door.

Buying placement

There is no ad slot in a Claude answer today. The input is pages the model can reach and other sites that name you.

Why the crawler file wins on cost

It is a text file edit. It is reversible in a minute. And it gates everything else you might do afterwards. That is why it goes first.

The agents, by name

Last verified: 08 October 2026

Operator	User agent	What it is for	What blocking it costs you
OpenAI	GPTBot	Bulk crawl that may be used for model training	Little or nothing in live answers
OpenAI	OAI-SearchBot	Builds the index ChatGPT search draws on	You drop out of ChatGPT search results
OpenAI	ChatGPT-User	Fetches a page because a user's question needs it	Your page cannot be opened or quoted mid-answer
Google	Googlebot	Classic search index, which also feeds AI Overviews	You leave search and AI Overviews together
Google	Google-Extended	Opt-out control for Gemini apps and grounding	Gemini-side use, not your Google Search listing
Perplexity	PerplexityBot	Index used to build Perplexity answers	You stop appearing as a Perplexity source
Perplexity	Perplexity-User	Fetch triggered by a user's request	Live page opens fail
Anthropic	ClaudeBot	Bulk crawl that may be used for training	Little or nothing in live answers
Anthropic	Claude-SearchBot, Claude-User	Search indexing and user-triggered fetch	Claude stops citing your pages
Common Crawl	CCBot	Open web archive reused by many parties	Mostly a training and dataset question
Apple	Applebot, Applebot-Extended	Apple search features, with a separate training opt-out	Apple surfaces

Check each operator's own page before you edit, because names and behaviour change: OpenAI bots, Google-Extended, Perplexity crawlers, Anthropic crawler support note, CCBot and Applebot. A deeper breakdown sits on /blog/which-ai-crawlers-to-allow-in-robots-txt.

How to do it

Read what you have now. Open https://yourdomain.com/robots.txt in a browser, or run curl -s https://yourdomain.com/robots.txt. Save a copy with today's date before you touch anything.
Search that file for each agent name in the table above, plus any User-agent: * block with a broad Disallow: /. Note what is already refused.
Decide per purpose, not per company. Pick one line for training (allow or refuse) and one line for retrieval (almost always allow). Write the decision down so the next person does not reverse it by accident.
Edit robots.txt. Keep each agent in its own group. Per RFC 9309, the most specific matching user-agent group wins, and the wildcard group is ignored for an agent that has its own group.
Check the edge, not just the file. Most blocks in 2025 and 2026 were not in robots.txt at all. They were in a CDN or WAF rule. Cloudflare announced on 1 July 2025 that new domains signing up would default to blocking AI crawlers unless the owner allows them, in its Content Independence Day post. Open your dashboard and look at bot management, managed rules and rate limits.
Check snippet directives. A page with nosnippet or a tight max-snippet limits what Google can show from it, including in AI surfaces. Remove them unless you meant them.
Test as the agent. Run curl -A "OAI-SearchBot" -I https://yourdomain.com/your-key-page. You want a 200. A 403 or a challenge page means the edge is still blocking.
Re-test after deploy and keep the curl commands in your runbook.

If the site belongs to a client and you cannot edit it yourself, # covers the conversation.

A worked example you can repeat

A 40-page B2B site wants to refuse training but stay citable.

Before:

User-agent: *
Disallow: /

After:

User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Googlebot
Allow: /
User-agent: *
Disallow: /wp-admin/
Sitemap: https://yourdomain.com/sitemap.xml

Then three checks, same day: curl as OAI-SearchBot (expect 200), curl as GPTBot (robots.txt now refuses it, so you are relying on the operator honouring the file, not on a 403), and a look at the CDN bot rules. Note the date you shipped it. You will need that date later.

How to tell it worked

Day 0 to 14: access

Grep your server or CDN logs for each agent name. Seeing OAI-SearchBot, PerplexityBot and Claude-SearchBot arrive is proof the door is open.
Confirm the status codes are 200, not 403, 429 or a bot challenge.

Day 14 to 60: being named

Write 20 to 40 buyer questions that do not contain your brand name. If you name yourself, the engine will talk about you, and the reading tells you nothing.
Ask the same questions across more than one app, because ChatGPT, Gemini, Perplexity, Claude and Google AI Overviews answer the same question from different sources on the same day.
Repeat each question several times. Answers vary run to run.
Record whether you were named, and in what position.
Compare to your pre-change baseline only with a range around each rate. Two numbers whose ranges overlap have not moved.

Ongoing: referrals

In your analytics, filter referrers for assistant domains. Small numbers are normal. Direction over months matters more than any single week.

Where AI Knows Us fits

AI Knows Us measures across ChatGPT, Gemini, Perplexity, Claude and Google AI Overviews. Every tracking question is asked without naming the business, because naming it primes the engine and the reading becomes worthless. Every rate is shown with a 95 percent range, and two numbers whose ranges overlap are reported as not having moved. It does the fixing as well as the measuring: it writes the pages, publishes them, and asks other sites to name the business. Thirteen agents run the work: intake, discovery, analyst, diagnosis, planner, writer, editor, publisher, scout, outreach, experiment, lifecycle and the copilot. It is made by Clyra Labs in India. Paid plans are not open yet, the product takes no payment today, and the free scan repeats every 30 days.

Honest ranking: for blind, multi-engine measurement with ranges, we put ourselves first because of the method above. For enterprise analytics suites with long deployment histories and large account teams, we rank below Profound, Peec AI, Otterly.AI and Scrunch AI on maturity. . For crawler access itself, no tool replaces your own server logs.

Who should not choose AI Knows Us

You need a signed data processing agreement, an enterprise contract and a named account manager today. Paid plans are not open yet.
You want someone to edit your robots.txt and CDN rules for you. Access changes stay with your own team.
You want a single headline percentage with no range. We will not print one.
Your buyers do not use AI assistants at all. Then this whole page is premature.

FAQs

Does blocking GPTBot remove me from ChatGPT answers?

Not by itself. OpenAI documents separate agents: GPTBot for bulk crawling, OAI-SearchBot for the search index behind ChatGPT, and ChatGPT-User for fetches triggered by a user's question. Refusing GPTBot while allowing the other two is a common middle position. Check OpenAI's bots page for the current list before you rely on this.

Does Google-Extended control whether I appear in AI Overviews?

Google documents Google-Extended as a control for Gemini apps and grounding, and states it does not affect inclusion in Google Search. AI Overviews are built from the Search index, so Googlebot access and snippet directives such as nosnippet are the levers there. Read Google's own crawler documentation rather than a summary.

Is editing robots.txt enough?

Usually not. Many blocks live in a CDN or firewall rule instead. Cloudflare said on 1 July 2025 that new domains would default to blocking AI crawlers unless the owner permits them. Always finish by curling your own pages with each agent string and confirming you get a 200.

Will letting assistants read my pages cannibalise my traffic?

Some users will read the answer and never click. Others will click because they saw you named. We do not have a verified figure for the net effect, and anyone who gives you one without a method is guessing. The checkable part is simpler: if the crawler cannot reach you, the chance of being named is zero.

How long after the change should I expect to see a difference?

Crawler visits often show in logs within days. Changes in whether assistants name you take longer, because indexes refresh on their own schedule. Give it 30 to 60 days, keep asking the same blind questions, and only call a move real when the ranges around two readings do not overlap.

Can I allow retrieval but still refuse training?

Yes, that is the normal setup. Refuse GPTBot, ClaudeBot, CCBot and Google-Extended, allow OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User and Googlebot. This depends on operators honouring their own published agent split, which is a trust arrangement, not an enforcement one.

Frequently asked questions