A publisher that blocks eleven of fourteen AI agents and allows exactly two

A publisher deciding about AI agents does not have to decide once. theverge.com shows what the alternative looks like when somebody has actually done the work. Its robots file blocks 11 of the 14 named agents we tested at the root and explicitly allows two of them, GPTBot and OAI-SearchBot. The fourteenth, OAI-AdsBot, appears nowhere in the file at all. At the edge, measured separately, the same host returns HTTP 403 to PerplexityBot and ClaudeBot and 200 to the rest, including all three of OpenAI's fetchers. Two layers read on two dates, pointing the same way.

That internal consistency is what makes it worth a page. A file and an edge that agree is a sign that one person, or one team with a written position, configured both. Most of what you find when you read other people's configurations is drift, where the file says one thing because of a decision in 2023 and the edge says another because of a setting nobody remembers enabling.

What the file blocks and what it lets through

The eleven blocked at the root are the ones a publisher would block if the question in front of them was bulk collection and third party reuse: ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Applebot-Extended and meta-externalagent. The two allowed by name are the two that feed OpenAI's crawl and the search index behind its answers.

The fourteenth is the detail worth pausing on. OAI-AdsBot is not blocked, and it is not allowed either, because it is not in the file. OpenAI documents four crawler identities and across the 24 readable robots files we parsed on 20 October 2026, not one of them names that fourth identity, while nine name GPTBot. A name that exists in the documentation and in nobody's rules is not a loophole anybody is exploiting; it is a list that was copied before it was finished.

The edge reading, and why it corroborates rather than repeats

The access run was a separate exercise with a separate method: 16 public urls requested under 11 agent names, 176 attempts, 23 refused outright, then a confirming reading taken hours later from another network. This host was one of only two whose per agent differences survived that second reading.

Two things make its rows trustworthy. The refusal was a plain forbidden page with no browser challenge in it, which separates it from the pattern where a client fails a JavaScript check and receives a 403 that means nothing about policy. And the direction matches the file: the two agents refused at the edge are two of the eleven the file disallows. When the permission layer and the enforcement layer say the same thing about the same names, you are looking at a decision rather than an accident.

Agent	In the robots file, 20 Oct 2026	At the edge, 16 Oct 2026
GPTBot	Allowed by name	200
OAI-SearchBot	Allowed by name	200
ChatGPT-User	Blocked at root	200
PerplexityBot	Blocked at root	403
ClaudeBot	Blocked at root	403
OAI-AdsBot	Not named	Not in the access run

Row three is the honest wrinkle and it belongs on the page. The file disallows ChatGPT-User and the edge served it. Several operators document that a fetch triggered by a person asking a question is a different kind of event, and that robots rules may not apply to it. A file saying no and a request arriving anyway is the documented behaviour rather than a violation, and it is the single most misread row in this whole subject.

The cost side, which this host evidently accepted

Blocking is not free, and the measurement is available. In our own citation audit of 16 October 2026, across 1,260 readings of ChatGPT and Gemini, theverge.com was cited in 197 of them, making it the only news publisher in the top ten domains of that run. News or large publisher sources took 3.2% of all citations. So a publisher with real standing in those answers has chosen to refuse most of the agents in the category and has kept the two that feed the engine where it is most often read.

Whether that trade is right for you depends on what your pages are for. An archive you licence is a different asset from a service page whose job is to be quoted back to a buyer. The useful move is to write down which of your sections is which before you touch a single rule, because the file is per path as well as per agent and most sites need both halves.

How to write a per operator policy instead of a switch

• Group the names by job, not by company. Bulk crawlers that gather text at scale, index builders that feed an assistant's own search, and fetchers that arrive because one person asked one question. Those three decisions have different consequences.
• Name every agent explicitly rather than relying on a wildcard. Under the Robots Exclusion Protocol the most specific matching group wins and the wildcard group stops applying, so one named group changes that agent's permissions across the whole site.
• Decide per path. A paid archive and a pricing page rarely deserve the same rule.
• Check the edge afterwards, because that is the layer that refuses requests. A file is a request and a firewall is an answer.
• If you want a page crawled but not quoted, the lever is a snippet directive rather than a crawl block, and every value is set out in our guide to the robots meta tag and when to use the header instead.

Questions about selective blocking of AI crawlers by a news publisher

Why allow OpenAI's crawlers and refuse the others?

We cannot speak to anybody's reasoning and will not invent it. What the configuration does is keep the host present in the index behind one set of answers while refusing the agents whose documented job is collection, which is a coherent position a publisher could defend.

Is blocking most AI agents bad for a publisher?

It removes you from answers drawn by the agents you refuse and it changes nothing about the ones you allow. The figure to put against it is how often your domain is cited in the answers you care about today, which is a measurement rather than an opinion, and our page on what a search console export and a visibility tracker each measure sets out what each tool can and cannot tell you.

Does a root block stop a user triggered fetch?

Not reliably, and several operators say so in their own documentation. That is why this host's file disallows one of those fetchers while the edge still served it, and why a rule you actually want enforced belongs at the edge.

How do I know a 403 here is policy and not a challenge?

Read the body. A plain forbidden page with no request to enable scripts or cookies is a refusal. A page asking you to pass a browser check is something your client failed rather than something the host decided about your name.

Can I tell any of this from the outside?

Yes, and that is the point of publishing it this way. Both layers of this reading are public: fetch the robots file, then fetch a page under each agent name and record the status. Two readings hours apart give you an answer you can stand behind.

A configuration is a statement, so read it as one

The reason to read somebody else's rules is not to copy them. It is to see that the decision has more than two settings, and that a publisher who thought about it ended up with eleven refusals and two deliberate exceptions rather than a single toggle. We publish our own readings in the same shape, with the dates and the denominators attached, and the two Google Search Console exports and the dated blind scan behind our claims sit on our proof page for anybody who wants to check the arithmetic before believing the method. Measuring this on client domains, and then fixing what it shows, is the service we charge for.