Which AI crawlers to allow in robots.txt

The problem this solves

You opened your robots.txt file because someone said AI assistants were sending traffic, or because a security plugin offered to "block AI scrapers" and you were not sure what that would cost you. Both are real worries. The file is small, it is public, and one wrong line can quietly take you out of answers for months before anyone notices.

The confusion comes from the names. "GPTBot" sounds like one switch for ChatGPT. It is not. OpenAI, Google, Perplexity and Anthropic each run several bots with different jobs. One crawls to build an index. One fetches a page because a user just asked about it. One gathers training data. They are controlled by different user agent strings, and blocking the wrong one has effects you did not intend.

The crawlers, one row each

Each user agent below is taken from the operator's own documentation. Links are in Sources at the end.

Why this step and not another

People reach for robots.txt first because it is the cheapest change on the list. It is one text file, it takes ten minutes, and it is the control every one of these operators says they read. Compared with rewriting pages or chasing mentions on other sites, it is almost free.

But understand what it is. Robots.txt is a request, not a lock. RFC 9309, the standard that defines the file, describes it as a way for a service to indicate access rules that well-behaved crawlers follow. It does not stop anyone. If your content must not be fetched, you need authentication, a paywall or firewall rules, not a text file.

Three more reasons this step comes before the others:

1. It can be silently wrong. A single Disallow: / under the wrong group can cost you every AI citation you would otherwise earn. Fixing it costs nothing.
2. Your CDN may already be blocking. Cloudflare announced on 1 July 2025 that new domains on its network default to blocking AI crawlers unless the owner opts in. Your robots.txt can say Allow while the edge returns a 403. Check both.
3. Group matching is counter-intuitive. Under RFC 9309, a crawler obeys the most specific matching group and ignores the others. So the moment you add a User-agent: GPTBot group, your User-agent: * rules stop applying to GPTBot. People add an Allow line for GPTBot and accidentally open /checkout/ to it.

Robots.txt is not the whole job. It controls fetching. It does not control snippets. If you want a page crawled but not quoted, use the robots meta tags: nosnippet, max-snippet, or data-nosnippet on a block of HTML.

How to do it: exact steps

1. Open https://yourdomain.com/robots.txt in a browser. Confirm it returns HTTP 200 and plain text, not an HTML error page.
2. Save a copy of the current file with today's date in the filename. This is your rollback.
3. Read the existing User-agent: * group. Write down every Disallow line in it. You will need to repeat them.
4. Decide your position in one sentence. Most businesses selling a service want to be found and quoted. Publishers selling access usually do not.
5. If your answer is "allow everything that is already public", the simplest correct file names no AI bots at all. Keep one wildcard group and let it cover them.
6. If you want to be explicit, write one group per agent and repeat your private-path Disallow lines inside each group. Do not assume the wildcard group carries over.
7. Add your sitemap line at the end of the file.
8. Upload, then reload the live URL and read it back character by character. Many CMS platforms serve a generated file and ignore the one you uploaded.
9. Check your CDN or WAF for a separate "block AI bots" setting and align it with the file.
10. Put a dated comment at the top so the next person knows when it was reviewed.

Worked example you can copy

A B2B software company, example.com, wants AI assistants to read its product and pricing pages, wants nothing crawling the customer account area, and wants to stay in Google Search.

The wrong version, which opens the account area to OpenAI:

User-agent: *
Disallow: /account/

User-agent: GPTBot
Allow: /

GPTBot matches its own group, so Disallow: /account/ no longer applies to it.

The correct version:

# robots.txt for example.com. Reviewed 2026-10-03.
# AI crawlers are welcome on public pages.

User-agent: *
Disallow: /account/
Disallow: /checkout/
Disallow: /search?

Sitemap: https://example.com/sitemap.xml

That is the whole file. No AI bot is named, so every one of them falls into the wildcard group, gets the public site, and is kept out of /account/. Fewer lines, fewer mistakes.

If a lawyer or an editor insists on named groups, repeat the Disallows:

User-agent: GPTBot
Disallow: /account/
Disallow: /checkout/
Allow: /

Then copy that block for OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Claude-SearchBot, Google-Extended and Applebot-Extended.

How to tell it worked

Do not trust the edit. Verify it. Five checks, in order of how fast they pay off.

1. Read the live file. Fetch it with a plain request, not a logged-in browser session. Confirm 200 and text/plain.
2. Grep your access logs for the user agent strings over the next 7 to 14 days. Search for GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot. Count hits per agent per day. Zero hits for an allowed agent after two weeks means something upstream is blocking, usually the CDN.
3. Check the status codes, not just the hits. A run of 403 or 429 responses to a named AI crawler means your firewall is overriding the file.
4. Confirm the bot is real. User agent strings are trivially faked. OpenAI publishes IP ranges as JSON at openai.com/gptbot.json, openai.com/searchbot.json and openai.com/chatgpt-user.json. Google publishes equivalent lists. Match the requesting IP against them before you celebrate or panic.
5. Ask the assistant. In ChatGPT or Perplexity, paste one of your URLs and ask it to summarise the page. If it reports it cannot access the page, your allow is not working. This is a live test of the user-triggered fetchers, which is the path most buyers actually trigger.

One honest caveat on expectations. Getting crawled is not the same as getting recommended. Access is the floor, not the outcome. Once the crawlers can read you, the next question is which pages are worth writing: see /blog/how-to-pick-which-pages-to-write-first. For a dated access control test across engines, see #.

Where AI Knows Us fits

Robots.txt is a one-time access fix. The harder question is whether assistants actually name you when a buyer asks for a recommendation, and that needs measuring over time.

AI Knows Us measures across ChatGPT, Gemini, Perplexity, Claude and Google AI Overviews. Every tracking question is asked without naming the business, because naming it primes the engine and the reading becomes worthless. Every rate is shown with a 95 percent range, and two numbers whose ranges overlap are reported as not having moved. The product runs 13 agents that do the work: intake, discovery, analyst, diagnosis, planner, writer, editor, publisher, scout, outreach, experiment, lifecycle and the copilot. It is made by Clyra Labs in India.

On pricing, the plain facts: paid plans are not open yet, the product takes no payment today, and the free scan repeats every 30 days.

Who should not choose AI Knows Us

• You only wanted the robots.txt answer. Copy the file above and close the tab. You do not need a product for this.
• You need a signed contract and an invoice today. Paid plans are not open yet.
• You want log-level crawler analytics or bot firewall management. That is a CDN and server logs job, not what is described here.
• You want a tool that names your brand in its test prompts so the score looks better. The measurement approach here deliberately does the opposite.

Other tools in this category include Profound, Peec AI, Otterly.AI and Scrunch AI. Check each one's own site for current features and prices before comparing.

FAQ