One help site allows all fourteen agents in robots.txt and refuses five of them at the edge

A robots.txt allow is a statement of intent and an edge rule is what actually happens, and help.openai.com is the cleanest measured example of the two disagreeing. Its robots file allows every one of the fourteen named AI agents we tested, with no disallow at the root for any of them. Its edge returns HTTP 403 to GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot and to a request that sends no agent string at all, while serving a desktop Chrome string normally. Three of the five refused crawlers are run by the company that owns the site. Both readings were confirmed, the file on 20 October 2026 and the access behaviour on 16 October 2026 with a second reading taken hours later from another network.

That pairing is the whole point. Most writing on this subject tells you to check your robots file, and most of it stops there. Here is a host where the file is as permissive as a file can be and the requests still do not get through. If you only read the file you would conclude that everything is fine, and you would be looking at the wrong layer.

What each layer says, measured separately

Layer	Read on	What it says
robots.txt	20 October 2026	No root disallow for any of the fourteen agents tested
The edge, named AI agents	16 October 2026, confirmed on a second reading	403 to GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot and ClaudeBot
The edge, no agent string	Same readings	403
The edge, desktop Chrome	Same readings	200

Read the bottom two rows together. A request carrying no name is refused and a request carrying a browser name is served, which tells you the decision is being made on the agent string by something that sits in front of the application. A permission file never had a vote.

Why this is probably not a decision anybody made about those names

The refusal body is the part worth reading, and it changes the story. It asks for JavaScript and cookies, which is the signature of a browser challenge rather than a policy page. A plain client cannot supply either, so it fails the check and receives the refusal every time, reproducibly, which is why the row held on both readings.

So the honest description has two halves. The behaviour is confirmed: these agents do not get the page, and that is stable across sessions and networks. The cause is not established, and we are not going to assert that anybody chose to refuse their own crawlers. A bot management product doing its default job produces exactly this result without a human ever naming an agent. Both statements are useful to a site owner and only one of them is a finding about intent.

The host doing this is one of the most cited domains in the category

Here is the part that makes it more than a curiosity. In our own citation audit of 16 October 2026, across 1,260 readings of ChatGPT and Gemini, help.openai.com was cited in 269 of them. It sits fifth on that list. A domain that a plain fetcher cannot read is among the most quoted sources in its own subject area.

That is a useful corrective to a claim this category repeats too easily. Being fetchable by named AI agents on a Tuesday afternoon is not what earns a citation. Official platform documentation took 12.3% of all citations in that run and vendor or agency own sites took 82.0%, and those sources win on being the operator's own documentation rather than on their delivery behaviour. Access work buys you the chance to be read. It does not buy authority, and the distinction is worth holding onto when somebody sells you a crawler access score.

The same inversion happens on ordinary sites, usually by accident

You do not need to run a help centre for this to apply. The standard way a marketing team ends up invisible is that somebody enabled a bot protection setting, the robots file was never touched, and nobody fetched a page under an agent name afterwards to see what came back. The file and the edge are owned by different people and reviewed on different schedules, which is the real mechanism.

• Fetch your own robots file as a plain client and record what it says per agent, with the date.
• Then fetch three real pages under each agent name you care about and record the status of every hop, not only the last one.
• Where those two readings disagree, the edge is the one to fix, and the setting is usually in your CDN or bot management dashboard rather than in anything a content team can reach.
• Read the body of every non-200 before writing it down. A challenge body and a forbidden page need different conversations with different teams.
• Take the second reading hours later. A refusal that does not repeat was never a refusal.

The operator documentation for each agent name, the version strings that break a naive filter and the published address ranges for verifying a request are collected in our guide to logging and monitoring AI crawler visits.

Questions about a help centre refusing named AI crawlers

Does a 403 under GPTBot mean the site has blocked OpenAI?

Not on one reading. It means a request carrying that string was refused once, and the two things it could be, a rule about the name or a challenge the client cannot pass, look identical in a terminal. Read the body and take a second reading in a separate session before you describe it as a block.

Why would a company refuse its own crawlers?

Usually nobody decided to. Edge protection is configured for abuse and scraping, the named AI agents arrive looking exactly like the traffic it was bought to stop, and the robots file that says allow is a separate system with a separate owner.

If the robots file allows a crawler, is the site not obliged to serve it?

The file is a request to well behaved crawlers about what to fetch, not a commitment by the server about what it will return. Enforcement only ever lives in the server or the edge, which is why the two layers can say opposite things and nothing breaks.

Does a request with no agent string tell me anything useful?

It is one of the most useful arms of the test. If a nameless request is refused and a browser string is served, the decision is being made on the string. If both are refused, you are probably looking at something counting requests rather than reading names.

Should I copy the robots file of a site that gets cited a lot?

There is nothing to copy here. This file allows everything, and the host is still unreadable to a plain fetcher. What it gets cited for is being the operator's own documentation for its own product, which is a position rather than a configuration.

Reading both layers is the whole job

A single measurement on one layer produced the most confident wrong answer available on this subject, and the only thing that fixed it was going back and reading the other layer and the response bodies we already had. That is the habit worth taking from this page, and it is cheaper than it sounds: two fetches and a pause. Our own numbers, including the two Google Search Console exports and the dated blind scan that sit behind every claim we make, are published with their windows and export dates on our proof page. Finding this class of gap on a client domain and closing it is what we charge for, so the method is printed here in full rather than summarised. For the controls OpenAI itself publishes and what they do and do not cover, our page on what OpenAI documents about controls is the companion read.