Nobody in our source table blocks OpenAI's crawlers, so permission is not the gap

On 20 October 2026 we fetched /robots.txt from 26 domains that our own citation audits had named, followed every redirect, and parsed each file into per agent rule groups. 24 of the 26 were readable. Across those 24, the count of files blocking GPTBot at the root is zero. The count blocking OAI-SearchBot is zero. The count blocking OAI-AdsBot is zero. Whatever is keeping a business out of these answers, it is not a permission file belonging to anybody already in them.

That matters because the advice most often given about AI visibility starts at the permission layer. In the set of sites that actually get cited in our category, the permission layer is wide open and the competition is happening somewhere else.

The counts, agent by agent

Agent	Blocked at root, of 24	Allowed at root	Named explicitly in a file
GPTBot	0	24	9
OAI-SearchBot	0	24	7
OAI-AdsBot	0	24	0
ChatGPT-User	1	23	8
ClaudeBot	1	23	9
Claude-User	1	23	6
Claude-SearchBot	1	23	6
Google-Extended	1	23	6
PerplexityBot	1	23	9
Perplexity-User	1	23	4
CCBot	2	22	8
Applebot-Extended	2	22	8
meta-externalagent	2	22	6
Bytespider	6	18	9

Every single one of those rows is checkable by anybody with a browser, because a robots.txt is a public file. That is unusual for a first party result and it is the reason this reading is worth publishing.

Which 26 domains, and why that selection

The sample is not a convenience list. It is the field our own measurements put us in.

• The ten most cited domains of the 16 October 2026 run, which drew 20,143 citations across 4,947 distinct domains from 1,260 readings.
• The six named specialist competitors our earlier run counted individually.
• The adjacent search suites and the directories our data surfaced.
• Four small agencies the earlier run named as winners in their own markets.
• Our own domain, because a sample that excludes the party publishing it is worth less.

Two of the 26 returned HTTP 403 and are reported as not usable rather than as blocks. One of them is otterly.ai, and a 403 from that host has already been shown in this programme to be a rate limiter rather than a policy. Calling an unreadable file a block would have been the easy way to a louder finding and a wrong one.

What this moves to the top of a diagnosis

If permission is not the common failure, the layers underneath it are where the work is. Four of them, in the order they actually break.

• Delivery. Whether your server, CDN or bot manager returns a 200 to a request carrying that agent name. In our own access run one host allowed every bot name in its file and still refused five named crawlers at the edge, confirmed on two readings hours apart.
• Content in the first response. Whether the facts are in the bytes that come back rather than written in afterwards by a script.
• Correctness. Whether what came back is the page you meant, rather than a consent wall or a regional variant.
• Evidence. Whether the page, once read, carries anything specific enough to be quoted in an answer.

The first three are configuration and are usually a day's work. The fourth is writing, and it is where the time goes. The protocol for testing the first three on your own domain, including the arm that catches a page whose text appears only after scripts run, is published as a dated test of robots.txt, HTTP access and named agents. Whether to allow a given crawler at all is a separate decision with its own measurement, set out in a dated robots.txt and access control test.

The edges of this reading

It is 26 domains on one date, chosen because our own audits named them, which makes it a reading about this category and not about the web. It records what each file permits at the root and not what happens deeper in a site. A permission is not an access grant, and the clearest proof of that is inside the same dataset, where a host that allows all fourteen agents in its file refuses several of them at the edge. No paywall, cookie banner or consent wall was measured anywhere in it.

We sell the work of getting a business read and quoted correctly, so our own site is in the sample and our own citation count from the audits is printed too: 6 readings of 1,260. The Google Search Console side of our own record, with both windows and the export date, is on our proof page.

What a result like this is worth, and why we published it today

Most claims in this field cannot be checked by the person reading them. This one can, by anybody, in a browser, in about ten minutes, which is unusual enough to be the main reason it is worth printing.

• The input is public. A robots.txt is served to everybody, so every row above is reproducible without access to our data.
• The sample is stated rather than described. Twenty six domains chosen by five named criteria, with the count that was readable printed beside the count that was fetched.
• The parsing rule is the part that decides the answer. We read the rule group that applies to each named agent at the root, and a reading that merges a site's groups together gets a different and wrong answer.
• It will go stale. These are files their owners can change this afternoon, so the date on this page is doing real work and the reading needs taking again.

That last point is the reason a sentence restating somebody else's policy carries the date it was read. A permission file is a live document, and a page quoting one without a date has made a claim about the present using evidence from the past.

Questions about robots.txt across the domains that get cited

How many of the sites you tested block GPTBot?

None of the 24 readable files, read on 20 October 2026. The same is true of OAI-SearchBot and OAI-AdsBot.

Which agent is refused most often?

Bytespider, by 6 of the 24, which is more than twice as many as any assistant agent in the sample.

Does an allow in robots.txt mean a crawler can read the page?

No, and this dataset contains the counterexample. A file can permit every agent while the infrastructure in front of it returns a refusal, and only one of those two answers the request.

Why report two domains as unreadable rather than as blocks?

Because a 403 on a single reading has already turned out to be a rate limiter twice in this programme, once with an explicit HTTP 429 behind it. An unreadable file is an unanswered question, not a finding.

Which audits chose the 26 domains?

Our two citation runs of 15 and 16 October 2026 on ChatGPT and Gemini, plus the city scan. The first of those, with its full method and source mix, is published as 597 readings across ChatGPT and Gemini in three markets.

Stop auditing the door and start auditing the room

The door is open at every one of the 24 sites we could read. So when a diagnosis of your own absence begins and ends with your robots.txt, it has checked the layer least likely to be wrong. Fetch your own page under each agent name, read the bytes, and then read the page as a stranger looking for something to quote. The gap is almost always in one of those two places, and both are yours to close.