The help centre cited in 269 readings refuses the crawlers its own robots file allows

help.openai.com was a source in 269 of the 1,260 readings we took from ChatGPT and Gemini on 16 October 2026, which puts an operator's own help centre in the top five of the source table. On 20 October 2026 we fetched the robots.txt of 26 domains our audits had named, and that host allows every one of the fourteen agents we tested. On 16 October 2026, across the access run, the same host returned HTTP 403 to GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot and to a request carrying no agent at all, and served desktop Chrome. Both readings of the 403 agree, taken hours apart.

So one of the most cited hosts in our data publishes a permission file that says yes and an edge that says no. The practical lesson transfers to any site: a robots.txt allow is a statement of policy, and the thing that answers the request is the infrastructure.

The two measurements, side by side

What was measured	Date	Result for help.openai.com
Readings citing the host, of 1,260	16 October 2026	269
Agents blocked at the root in robots.txt, of 14 tested	20 October 2026	0
HTTP status under five named AI crawler agents	16 October 2026, confirmed on a second reading hours later	403
HTTP status under desktop Chrome	Same run	200
HTTP status under no agent string at all	Same run	403

The last row is the one that settles what kind of control this is. A rule aimed at AI crawlers by name would let an anonymous request through. This one does not, which points at a bot management layer reacting to the shape of the request rather than to a list of names.

Why the host still reaches 269 readings

Being refused at one edge on one date does not remove a page from an answer. Material gets into an answer through more than one route, and the citation count records which domain the answer was assembled from rather than which request succeeded last Tuesday. For a publisher the useful reading is narrower and more useful: the citation count and the access check answer two different questions, and a plan built on only one of them is built on half the picture.

• The citation count tells you who is in the field. It is the competitive picture.
• The access check tells you whether your own pages can be fetched. It is the plumbing, and it is yours to fix.
• They fail for different reasons. Absence from a source table is usually an evidence problem. A 403 is a configuration problem, and it is faster to fix.

The rule we now apply to every 403 we record

This one is paid for in retractions. Two hosts in the same access run looked like per agent blocks and were not: both returned 200 with the full article hours later to almost every agent, Chrome included, and one of them returned an explicit HTTP 429. They were rate limiters. A reversed order pass on the same day did not reveal it, and 20 of the 23 recorded refusals carried a browser challenge body rather than a refusal.

• A 403 counts only when it repeats in a separate session hours later. Three readings, not two, and the third one spaced out.
• A 429 anywhere in the set puts the whole set in doubt. It is the host telling you what the other statuses were.
• Read the saved bodies, not only the statuses. A challenge page is not a policy.
• Save every response body so the question is answerable later without running the fetches again.

The help centre finding survived that test on both readings, which is why it is printed here. The two that did not survive are withdrawn in public, with the 429 given as the reason.

Running the same check on your own site

Fetch your own pages under each named agent, record the status, and read the bytes that came back. A 200 carrying a consent wall is a failure. A 200 carrying the article with your prices missing because they are written in by a script is also a failure. The full protocol, including the arms that catch each of those, is published as a dated test of robots.txt, HTTP access and named agents. What the operator publishes about its own crawlers, and the parts worth keeping in your notes, sits alongside the controls described in the controls and what OpenAI publishes.

Our own site passes this check and we publish the result rather than asserting it: the benchmark page returns the identical article to all eleven agents tested, 63KB and 1,789 words, under GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Googlebot, Google-Extended, curl and a request with no agent. We sell the work of fixing this for other sites, and the Search Console side of our own record is on our proof page.

Questions about the 269 readings and the 403

How often was help.openai.com cited?

In 269 of 1,260 readings on 16 October 2026, behind developers.google.com at 446, support.google.com at 328, developers.google.cn at 325 and blog.google at 309. The earlier reading in the same programme, with its full method and source mix, is at 597 readings across ChatGPT and Gemini in three markets.

Does its robots.txt block AI crawlers?

No. Across the 26 domain sample we read on 20 October 2026, that host allows all fourteen agents we tested at the root. The refusal happens after the policy file, at the edge.

Is a 403 the same as being blocked?

Only when it repeats in a separate session hours later and the response body is a refusal rather than a browser challenge. Two findings in our own access run failed that test and were withdrawn, one of them with a 429 on record.

What should I check first on my own site?

Delivery before permission. Most owners read their robots.txt, which is the layer least likely to be the problem, and never fetch their own page under a crawler agent name, which is the layer that usually is.

Which crawler names are worth testing?

OpenAI documents four, GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot, read from its own documentation on 16 October 2026. Anthropic documents ClaudeBot, Claude-User and Claude-SearchBot. Google lists Googlebot and Google-Extended, and Perplexity lists PerplexityBot and Perplexity-User. Any page restating that list carries the date it was read, because the list grows.

The sentence to take away from one host

A permission file is a promise about intent and the edge is the thing that answers. When a host as central as this one carries a yes in the file and a no at the door, the only reliable way to know what happens to your pages is to send the request and read what comes back. That is a half hour of work on your own domain, and it is the half hour we start with on every account we take on.