Our own robots.txt blocks none of the fourteen agents, and here is the whole file

Our own permission file, fetched on 20 October 2026, returns HTTP 200 as 418 bytes of plain text and blocks none of the fourteen named agents at the root. It names nine of them explicitly in a single group, and the other five fall into the wildcard group, which also permits the root. A company selling the argument that your pages should be readable by assistants has no business asserting that about its own site, so the file is printed below in full and the address is public, which means this page can be checked rather than believed.

User-Agent: GPTBot
User-Agent: OAI-SearchBot
User-Agent: ChatGPT-User
User-Agent: ClaudeBot
User-Agent: Claude-SearchBot
User-Agent: PerplexityBot
User-Agent: Google-Extended
User-Agent: Applebot-Extended
User-Agent: CCBot
Allow: /
Disallow: /app/
Disallow: /api/ User-Agent: *
Allow: /
Disallow: /app/
Disallow: /api/
Disallow: /verify Host: https://www.aiknowsus.com
Sitemap: https://www.aiknowsus.com/sitemap.xml

Two groups, two blocked directories, and a sitemap reference. The two paths kept out of both groups are the signed in application and the API, neither of which has a page anybody would want quoted. Every public page on the site is permitted to every agent in the list, and the access side of that claim was measured separately: our benchmark article came back identical to all eleven agent names tested, at 63KB and 1,789 words, published as the bytes we got back under eleven agent names.

Which of the fourteen the file names, and which it does not

Nine named, five not, none blocked at the root either way. The right hand column is the interesting one and we are going to be straight about it. OpenAI documents four crawler identities and this file names three, which puts our own site in exactly the group we counted in everybody else's: across the 24 readable files in that sample, nine name GPTBot and not one names OAI-AdsBot. A list of agent names is a restated document, and a restated document goes stale without anybody touching it. The counted version of that finding is nine robots files naming GPTBot and none naming the fourth OpenAI crawler.

The one path where the two groups differ, and why that is the lesson

Read the two groups against each other and one line is in the second and not the first: the wildcard group keeps everybody out of /verify and the named group does not mention it. Under the Robots Exclusion Protocol an agent is matched to one group, the most specific one naming it, and once that group applies the wildcard group stops applying to that agent entirely. So the nine named agents are permitted a path that every other client is kept out of, purely because they are named somewhere else in the file.

Nobody has to intend that for it to happen. It is the single commonest defect in this kind of file and it moves in both directions: add a named group to open one thing for one agent and every restriction in the wildcard group silently stops applying to it. That is why a file has to be read per agent by something that implements the group rule rather than by a person scanning it, and why the parser behind our 26 host reading self tests on a file containing a group with two agent names before it reads anybody else's.

What publishing your own file is actually worth

Not much as a technical statement, and a great deal as a disclosure. The permission layer is the layer least likely to be wrong in this category: not one of the 24 readable files we parsed refuses GPTBot, OAI-SearchBot or OAI-AdsBot at the root, so an open file makes a site unremarkable rather than exemplary. What it does is remove a question a sceptical reader is entitled to ask, at a cost of one URL.

• It is checkable in ten seconds. Any claim a reader can verify faster than they can doubt it is a claim worth making in public.
• It dates itself. This page says 20 October 2026, and a file is something its owner can change this afternoon, so the date is doing real work rather than decorating the page.
• It commits us to the gap as well as the strength. The five unnamed agents are on this page because leaving them off would have been the version written to flatter us.

Reading your own file to the same standard

• Fetch it as a plain client with redirect following on, and confirm it comes back as plain text with a 200. An HTML error page in that slot means nothing is reading the rules you think you published.
• Record a decision per agent, and record which group each one matched. Two agents allowed for different reasons are two different facts.
• Diff your named groups against your wildcard group line by line. Any path in one and not the other is a difference you either meant or did not.
• Re-read the operators' own documentation on a schedule and note on the page when a name was added, because the list grows and nothing tells you when it does.
• Then fetch a real page under each agent name, since a file is a request and a server is an answer. Elsewhere in our own sample a help centre permits all fourteen agents and refuses five of them at the edge on two separate readings.

What an open file still does not buy

It does not buy a citation, and our own numbers are the plainest evidence of that. With this file in place and every public page served identically to every agent we tested, our domain appeared in the source list of 3 of 597 readings on 15 October 2026 and 6 of 1,260 on 16 October 2026, both runs on ChatGPT and Gemini. Permission is the entry condition. What gets quoted turns on whether a page carries something specific enough to answer the question, which is writing rather than configuration, and what actually changes on a page when you work on that is set out in the rater framework and what changes on a page.

Our file, our gaps: what people ask about it

Is any named agent refused on this site?

No. Read on 20 October 2026, the file blocks none of the fourteen agents we test at the root. Two directories are kept out of both groups, the signed in application and the API, and neither holds a page intended for a reader.

Why name nine agents at all if nothing is being blocked?

The named group exists to state the position explicitly rather than leave it to a default, which is readable by a person auditing the file. The honest cost of doing it that way is in this page: a named list has to be maintained, and ours is already three of OpenAI's four rather than four.

Is a missing agent name the same as a block?

No. An agent the file does not name is governed by the wildcard group, and ours permits the root. Not being named and not being blocked are the same outcome here, which stays true only until somebody writes a restrictive default.

Which agents does the file leave unnamed?

OAI-AdsBot, Claude-User, Perplexity-User, Bytespider and meta-externalagent. All five are permitted at the root through the wildcard group, and the first of them is in OpenAI's own published documentation, read on 16 October 2026.

How does this compare with the other sites you read?

It is unremarkable, which is the point. Across 24 readable files from domains our audits named, the only host refusing assistant agents at the root was one news publisher, and the most refused agent in the whole sample was a bulk collector. The per agent and per host counts are in our counted reading of 26 robots files.

The file is 418 bytes and the argument is the address

Anything on this page can be falsified by one request to a public path, which is the only reason it is worth writing. Running that measurement for other companies, and then doing the writing that moves what an answer says, is how we earn a living, so the same rule applies to every number we publish about ourselves: the two Google Search Console exports behind our own claims, each with its window and the day the file was pulled, are printed on our proof page. A vendor that will not show you its own configuration is asking you to take the easy half on trust.