Nine robots files name GPTBot, none names the fourth OpenAI crawler

Twenty six domains, chosen because our own citation audits had surfaced them, had their permission files read on 20 October 2026 and parsed into the rule group that applies to each named agent. Twenty four of them answered. Of that 24, nine name GPTBot explicitly, nine name PerplexityBot, nine name ClaudeBot, eight name ChatGPT-User, seven name OAI-SearchBot and zero name OAI-AdsBot. The fourth OpenAI identity has been in the operator's own documentation since we read it on 16 October 2026 and it is in nobody's file in this sample.

That is not a scandal. It is the clearest measurable example of a problem every site owner has: a permission file written against a list of names is a restated document, and a restated document goes stale into a false claim without anybody touching it.

The naming counts, agent by agent

Agent	Named explicitly, of 24	Blocked at root
GPTBot	9	0
PerplexityBot	9	1
ClaudeBot	9	1
Bytespider	9	6
ChatGPT-User	8	1
CCBot	8	2
Applebot-Extended	8	2
OAI-SearchBot	7	0
Claude-User	6	1
Claude-SearchBot	6	1
Google-Extended	6	1
meta-externalagent	6	2
Perplexity-User	4	1
OAI-AdsBot	0	0

The last row is the one with the lesson in it. Nine of these sites went to the trouble of writing a rule that names one OpenAI crawler, and seven named a second, which means somebody sat down with the operator's list and worked from it. The list has since grown and the files have not.

Why a named agent list is a maintenance liability

Three separate things in this area change without notifying the people who copied them, and all three have already caught somebody out in our own programme.

• The list of identities grows. OpenAI documents four, GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot, read on 16 October 2026. Two of our own live pages said three until that reading, and one of them asserted a count out loud, which had become false.
• The agent strings carry versions that move. Read on 16 October 2026 they were OAI-SearchBot/1.4, GPTBot/1.4, ChatGPT-User/1.0 and OAI-AdsBot/1.0, and for two of them the documentation says the version number may change. A log filter pinned to a version will quietly stop matching.
• The published address ranges go stale at different rates. The four OpenAI range files hold 293 entries between them and every single one is IPv4. The creation stamps inside them are not close together: one carried a date from January while another was nine days old when we read them on 16 October 2026, so refreshing one is not refreshing another.

Put those together and a rule that names agents individually has three independent ways to drift out of date, none of which produces an error anywhere.

What to do instead, or alongside

• Put a date on the line. A comment in your robots.txt saying when the operator's list was last read turns a silent staleness into a visible one.
• Decide the policy, then express it. Most files should start from a default that applies to everything and name agents only where the decision genuinely differs. A file that is a list of names is a file that needs maintaining forever.
• Match on the stable part of the string. A filter looking for the product name rather than the product name plus a version number keeps working when the version moves.
• Re-read the operator pages on a schedule. Any page of yours that restates somebody else's list gets a date and a re-read interval, and when a new name appears you say on the page that it was added and when.
• Remember a file is not a door. In this same dataset one host allows all fourteen agents in its robots.txt and returns 403 at the edge to five named AI crawlers, confirmed on two separate readings. The two layers disagree and only one of them answers a request. The method for checking both is published as a dated test of robots.txt, HTTP access and named agents.

Where this reading stops

It covers 26 domains on one date, chosen because our own audits named them, which makes it a reading about this category rather than about the web. Two of the 26 returned HTTP 403 and are reported as not usable rather than as blocks, because a 403 from one of those hosts has already been shown in this programme to be a rate limiter. The counts describe the root of each site and not what happens deeper in it. No paywall, cookie banner or consent wall was measured anywhere in it.

We sell the work of getting a business read and quoted correctly by these agents, so our own domain is in the sample and our own citation figure from the audits goes on the page too: 6 readings of 1,260. The Google Search Console side of our own record, with both windows and the export date, is our proof page.

Questions about naming AI crawlers in robots.txt

How many of the files name OAI-AdsBot?

None of the 24 readable files, read on 20 October 2026, while nine name GPTBot and nine name PerplexityBot.

Does that mean OAI-AdsBot is blocked everywhere or nowhere?

Nowhere in this sample. Zero of the 24 block it at the root, which is the same count as GPTBot and OAI-SearchBot. Not being named and not being blocked are the same outcome here, and that is only true until somebody writes a default rule.

How many crawler identities does OpenAI document?

Four, as read from its own documentation on 16 October 2026: GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot. Any page restating a list like that carries the date it was read. The citation run that chose these 26 domains is published as 597 readings across ChatGPT and Gemini in three markets.

Should I write rules by name at all?

Write them where your decision genuinely differs between agents, and carry a default for everything else. A file that is only a list of names needs maintaining every time an operator adds one.

Why does a version number in the agent string matter?

Because two of the four documented strings say the version may change, so a log filter written against the full string including the version will stop matching without reporting anything.

Date the line, then the file can be audited

One comment with a date in it changes a robots.txt from a document nobody can assess into one anybody can. Twenty four files in this sample, nine of them written carefully enough to name a crawler, and not one of them says when its list was copied. That single line is the cheapest piece of maintenance available here, and it is the one that turns the other fourteen rows into something you can check next quarter.