Is that site blocking AI crawlers? We read 26 robots files and counted every rule

Mostly they are not blocking. On 20 October 2026 we fetched /robots.txt from 26 domains our own citation audits had named, followed every redirect, and parsed each file into per agent rule groups. 24 of the 26 were readable. Not one of those 24 blocks GPTBot, OAI-SearchBot or OAI-AdsBot at the root. Zero of twenty four. The agent refused by the most publishers in the set is not an assistant at all: it is Bytespider, blocked by 6 of 24. So when a named AI agent gets a forbidden response from a site, the robots file is the least likely explanation, and the check that most people run first is the one that almost never finds anything.

This is a first-party reading of other people's published files, which is why it can be rechecked line by line. Every file in it is public. Anybody can fetch the same 26 paths today and count the same groups, and if a publisher has changed its file since, the difference is the finding rather than a problem with the method. The date matters more than usual here, because a robots file is edited by whoever ships the next release.

The 26 domains, and why these 26

The set was not drawn at random. It is the list of domains that already matter to this category, assembled from measurements we had already published: the ten most cited domains of our second citation audit, the six specialist competitors our data named, the adjacent search suites and the directories our rows surfaced, four small agencies the audit recorded as winning their own cities, and our own domain. A robots file only tells you something useful when you know why that host is in the list.

Two of the 26 returned HTTP 403 to the request for the file itself. Those two are recorded as not usable, not as blocks, and the distinction is the whole discipline of this work. One of them is otterly.ai, and a forbidden response from that exact host has already been shown inside this programme to be a rate limiter rather than a policy: it served the full article to almost every agent on a later reading from another network. Counting those two as refusals would have produced a tidier table and a false one.

What the 24 readable files actually say, agent by agent

Three columns, because they answer three different questions. Blocked at root means the file disallows that agent from the top of the site. Named explicitly means the file has a rule group for that agent at all, whether it allows or disallows, which tells you how many publishers have thought about it.

Agent	Blocked at root	Allowed at root	Named explicitly
GPTBot	0	24	9
OAI-SearchBot	0	24	7
OAI-AdsBot	0	24	0
ChatGPT-User	1	23	8
ClaudeBot	1	23	9
Claude-User	1	23	6
Claude-SearchBot	1	23	6
Google-Extended	1	23	6
PerplexityBot	1	23	9
Perplexity-User	1	23	4
CCBot	2	22	8
Applebot-Extended	2	22	8
meta-externalagent	2	22	6
Bytespider	6	18	9

The fourth OpenAI identity is in the documentation and in nobody's file

Read down the right hand column and one row stands out. Nine of the 24 files name GPTBot and nine name PerplexityBot, so these are publishers who have sat down and made decisions about AI agents. Not one of the 24 names OAI-AdsBot. OpenAI documents four identities rather than the three most lists carry, and the fourth one appears in no rule group in this set.

That is not negligence on anybody's part. It is what a documentation list looks like after it has been copied into a hundred articles and before those articles are reread. The practical consequence is narrow and worth acting on: if you have written rules for OpenAI's crawlers, your file almost certainly covers three of four, and the names and jobs behind each one are set out in our guide to which AI crawlers to allow in robots.txt.

The most refused agent in the set is a scraper, not an assistant

Bytespider is blocked at the root by 6 of 24, more than twice any assistant agent in the table. The publishers doing it are recognisable: github.com, g2.com, capterra.com and rank4ai.co.uk each block Bytespider and nothing else of the fourteen. brightlocal.com blocks four, CCBot, Bytespider, Applebot-Extended and meta-externalagent, which is a coherent position about bulk collection rather than about answers.

One publisher is the exception that proves the pattern is deliberate. theverge.com blocks 11 of the 14 agents at the root and allows exactly two of them, GPTBot and OAI-SearchBot. That is a per operator policy written by somebody who knew the difference between the names. Every other readable host in the set blocked none of the fourteen.

Why your forbidden response is probably not coming from this file

Here is the measured case against checking robots first, and it comes from the same programme. help.openai.com allows every one of the fourteen agents in its robots.txt, and returns HTTP 403 at the edge to GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot and to a request carrying no agent string at all, confirmed on two separate readings hours apart. The policy file and the infrastructure disagree, and only one of them answers the request.

So the order of work inverts. Read your own file, because it is free and a wrong line in it is cheap to fix. Then go to the layer that actually refuses requests, which is your CDN, your firewall or your bot management product, and fetch your own pages under each agent name to see what comes back. A permission and a delivery are decided by two different systems, and the second one has the final word.

How to take this reading on your own domain

• Fetch your own /robots.txt as a plain client and confirm it returns 200 with a plain text content type. A challenge page or an HTML error in that slot is itself the finding.
• Parse it per agent rather than reading it with your eyes. Under the Robots Exclusion Protocol the most specific matching group wins and the wildcard group stops applying, so adding one named group changes what that agent is allowed to do everywhere.
• Write down all fourteen names and record allowed, blocked or partly blocked for each, with today's date beside it.
• Then fetch a real page under each name and record the status. A file that says allow and an edge that says forbidden is the most common shape of this problem, and the two readings together are the only pair that settles it.

Questions about reading robots files across other people's domains

Does a zero in the blocked column mean these sites welcome AI crawlers?

It means the file does not refuse them at the root on the day we read it. Access is decided further out, at the CDN or firewall, and a file that allows everything sits comfortably in front of an edge rule that refuses most things. Read both layers before deciding what a host has chosen.

Why record two hosts as not usable instead of counting them as blocks?

Because a forbidden response under an automated request is a reading that needs a second reading, and in this programme one of those two hosts has already produced a forbidden response that turned out to be a rate limiter. Twenty four is a smaller number than twenty six and it is the one the evidence supports.

Should I copy the rules from the publishers in this set?

Copy the shape, not the lines. The useful pattern here is that the publishers who decided anything decided it per agent and per job, and the single most blocked name belongs to bulk collection rather than to an assistant that can send a reader back to you.

How often does a robots file need rereading?

Put it on the same schedule as your release notes, because it changes when somebody ships. Our own copy of this reading carries 20 October 2026 on it for that reason, and a count without a date is already drifting.

Does any of this tell me whether an assistant will quote my pages?

No, and the two belong in separate records. Our own measured citation figure is 3 of 597 readings of ChatGPT and Gemini on 15 October 2026, and 6 of 1,260 readings of the same two engines on 16 October 2026, on a question set designed separately. Neither of those numbers moved because of a robots file.

What a counted robots file is worth, and what it is not

A file like this answers one narrow question completely: who has written down a decision about named AI agents, and what that decision says. It cannot tell you who gets quoted, and the gap between the two is where most of the money in this category sits. We publish our own worst number beside our method for exactly that reason, and the two Google Search Console exports and the dated blind scan behind our claims are all on our proof page. Running that measurement for other companies, and fixing the gaps it finds, is how AI Knows Us earns a living, which is why the 26 domains and the fourteen agent names are printed here in a form that needs nobody's permission to repeat. If you are weighing up doing this kind of work in house, the costs that are easy to miss are set out in our page on build or buy for an AI visibility stack.