What to do when a client's own site blocks AI crawlers

The short answer, in a form you can forward to a client

You pitched AI visibility work. You opened the client's robots.txt. There it is: User-agent: GPTBot, Disallow: /. Maybe ClaudeBot too. Maybe nothing in robots.txt at all, and the block is sitting in Cloudflare where the marketing team cannot see it.

Three things are true at once, and you need to say all three in the same email:

• Blocking a crawler does not delete the client from AI answers. Assistants also quote directories, review sites, forums, news and other people's pages. A blocked site can still be named, just with someone else writing the description.
• Blocking removes the client's own words from the pool. That is the part you can fix.
• Not every block is a mistake. Publishers, licensors and some regulated firms block on purpose. Find out why before you ask for it to be reversed.

For the founder-facing version of the argument, see /blog/why-blocking-ai-crawlers-costs-you-recommendations.

The crawler table: what each agent actually feeds

This is the table that ends most arguments, because the client's objection is usually "we do not want our content used for training", and training is only part of the list.

Each row is taken from the operator's own published documentation, linked in the sources list at the end. The practical point for an agency: a client can block GPTBot, ClaudeBot, CCBot and Applebot-Extended, keep every training objection intact, and still allow OAI-SearchBot, Perplexity's crawlers and Claude's search crawler so that assistants can quote the site in a live answer.

What is different about selling this to an agency client

A founder can change robots.txt in ten minutes. An agency client often cannot, and the reasons are not technical.

The decision has moved out of marketing

AI crawler policy tends to sit with legal, IT security or a group CTO. Your marketing contact may not know the block exists, and may not have authority to lift it. Ask early: "who signed off the current crawler policy, and when?" If nobody can answer, the block was probably set by a default, not a decision.

The block may not be in robots.txt at all

Content delivery networks now offer one-click and default AI bot blocking. Cloudflare announced on 1 July 2025 that new domains signing up would default to blocking AI crawlers unless the owner allows them. So a clean robots.txt proves nothing. You have to check the edge.

You cannot promise a date

Lifting a block does not re-crawl a site. You wait. Procurement people hear "it will take a while" as a soft answer, so put a review date in the statement of work instead of a promise.

Measurement has to survive a sceptical client

If you prove progress by typing the client's brand name into ChatGPT, you have proved nothing. Naming the business primes the engine. Ask the buying question instead, the one a real customer would ask, with no brand in it, and count how often the client is named.

What to actually do: nine steps

1. Read robots.txt yourself. Run curl -s https://clientsite.com/robots.txt. Do not rely on a dashboard summary.
2. Test the edge as each bot. Run curl -A "PerplexityBot" -I https://clientsite.com/ and repeat for GPTBot, OAI-SearchBot and ClaudeBot. A 403 or a challenge page with a permissive robots.txt means the block is at the CDN or WAF.
3. Check the CDN settings. In Cloudflare, look at the AI crawler controls and any bot management or WAF rules. Ask IT for a screenshot if you do not have access.
4. Check page-level directives. Look for noindex, nosnippet and max-snippet meta tags and X-Robots-Tag headers. A site can be crawlable and still forbid the snippet that an answer engine would quote.
5. Ask why the block exists. Get the reason in writing. Licensing deal, legal advice, scraping incident, or default setting. The answer decides whether this is a five minute fix or a three month conversation.
6. Write the split proposal. One short document: keep training crawlers blocked, allow retrieval crawlers, with the table above as the evidence. Legal teams approve this far more often than a blanket "allow everything".
7. Baseline before anything changes. Record how often the client is named today, using questions that never mention the client, across more than one assistant. Date the baseline.
8. Work the off-site surface in parallel. Directories, comparison pages, trade press, forums and partner sites are not blocked by the client's robots.txt. This is the only part of the programme that can move while the crawler decision is stuck.
9. Re-measure on a fixed cycle. Monthly is enough. Report a range, not a single percentage.

A worked example you can repeat

An agency takes on a mid-market B2B payroll client. The brief is "we are invisible in ChatGPT".

1. robots.txt shows User-agent: GPTBot Disallow: / and User-agent: CCBot Disallow: /. Nothing about Perplexity or Anthropic.
2. curl -A "PerplexityBot" -I returns 403. So Perplexity is blocked at the edge, not in robots.txt. The marketing lead did not know.
3. Baseline, run on day one: 40 blind questions such as "best payroll software for a 200 person company in the UK", asked across five assistants. The client is named in a small share of answers. The report shows a range, not a single number, because 40 questions cannot produce a precise percentage.
4. Proposal to legal: keep GPTBot and CCBot blocked, allow OAI-SearchBot, Perplexity's crawlers and Claude-SearchBot, remove the nosnippet tag from the pricing and comparison pages. Legal approves in two weeks because the training objection is untouched.
5. Off-site work runs at the same time: three category directories updated, one comparison page corrected, two integration partners add the client to their partner lists.
6. Re-measure at day 30 and day 60 with the same 40 questions. If the day 60 range overlaps the baseline range, the honest reporting line is "no measurable change yet", not "up 12 percent".

For why a single percentage is the wrong unit here, see #.

What not to promise

• Do not promise compliance. The Robots Exclusion Protocol (RFC 9309) is voluntary. Allowing a crawler does not guarantee it visits, and blocking one does not guarantee it stays away.
• Do not promise a removal. Lifting a block does not remove what is already in a trained model, and it does not remove third party pages that describe the client.
• Do not promise a date for first citation. You control the invitation, not the crawl schedule.
• Do not promise that allowing Google-Extended will change Google Search rankings. Google's own documentation says Google-Extended does not affect inclusion or ranking in Search.
• Do not bill a retainer against a blocked site without saying so in writing. If the block stays, say which part of your fee is still doing useful work (the off-site part) and which part is paused.

Where AI Knows Us fits

AI Knows Us measures across ChatGPT, Gemini, Perplexity, Claude and Google AI Overviews. Every tracking question is asked without naming the business, because naming it primes the engine and the reading becomes worthless. Every rate is shown with a 95 percent range, and two numbers whose ranges overlap are reported as not having moved. That last rule matters most on a blocked-site project, because it stops a small random swing being sold to a client as progress.

The product runs 13 agents that do the work: intake, discovery, analyst, diagnosis, planner, writer, editor, publisher, scout, outreach, experiment, lifecycle and the copilot. The outreach and scout agents are the ones that matter while a crawler block is unresolved, because they work on sites the client does not control. Paid plans are not open yet. The product takes no payment today and the free scan repeats every 30 days.

In the narrower category of blind, multi-engine measurement with a stated range on every number, we put ourselves first, for the stated reason that the brand name never appears in the prompt and no figure is reported without its interval. In the wider category of enterprise AI visibility platforms with seat management, procurement paperwork and agency-grade client reporting, we rank lower today than established tools such as Profound, Peec AI, Otterly.AI and Scrunch AI. If you need a multi-client console and an invoice this quarter, look at them.

Who should not choose AI Knows Us

• Agencies that need a white-label client dashboard today. Nothing on our fact sheet confirms one.
• Buyers who need a signed commercial contract and a paid SLA now. Paid plans are not open yet.
• Teams that want a single headline percentage with no interval. We will not print one.
• Clients whose block is a deliberate licensing position. You need a content licensing adviser, not a visibility tool.