The one news publisher in our top ten blocks eleven of fourteen agents and allows two

Exactly one news organisation reached the top ten of the source table we built from 1,260 readings of ChatGPT and Gemini on 16 October 2026, and it turns most crawlers away at the door. The host is theverge.com, and it held a place in the source list on 197 of those 1,260 readings. Four days later, fetching the robots.txt of 26 domains our audits had named, we found that host blocking eleven of the fourteen agents we tested at the root, with GPTBot and OAI-SearchBot allowed. The access run of 16 October 2026 had already recorded it returning HTTP 403 to PerplexityBot and ClaudeBot and 200 to the rest.

Two independent measurements, four days apart, pointing the same way: this is a deliberate policy written per operator, not a blanket position on AI. It is also the clearest example in our data of a publisher treating the question as a commercial one with different answers for different companies.

What the two readings found, in one place

Measurement	Date	Result
Readings citing the host, of 1,260	16 October 2026	197
Agents blocked at the root in robots.txt, of 14 tested	20 October 2026	11
Agents allowed at the root	20 October 2026	GPTBot and OAI-SearchBot
HTTP status under PerplexityBot and ClaudeBot	16 October 2026	403
HTTP status under the other agents tested	16 October 2026	200

In the same 26 domain sample, every other readable host blocked none of the fourteen except for five that each refused one or a handful: a code host, two directories, a small British agency and a local search tool blocked Bytespider, and one of those five also blocked CCBot, Applebot-Extended and meta-externalagent. This publisher is doing something none of the others in the sample does.

Why a publisher would choose per operator rules

A news organisation sells the thing being fetched. Its archive is inventory, which puts it in a different position from a software company whose documentation is marketing. Three consequences follow, and they are visible in the file.

• An allow can be a commercial relationship. Two names permitted out of fourteen is a shorter list than any accident would produce.
• A block is not a technical measure. It is a published position, and the 403 at the edge for two of the names shows the position is enforced rather than merely stated.
• The policy will move. It is a business decision, so a page restating it needs the date it was read, which is why both dates are printed above.

What this means if you were counting on press coverage

News and large publishers took 3.2% of the 20,143 citations in the 16 October run, and this single host accounts for the category's only appearance in the top ten. So the ceiling on a press route into a source list is set by two things at once: how small the category's share is, and whether the publication you landed in permits the agents you care about. The second one is checkable in a minute and almost nobody checks it.

• Before you value a placement, fetch that publication's robots.txt. A piece in a title that blocks the agents you are measuring will not reach those answers.
• Check the edge as well as the file. The two layers disagree often enough that one reading is not an answer, and the method for both is published as a dated test of robots.txt, HTTP access and named agents.
• Keep the coverage anyway if people read it. A placement a buyer reads has value that has nothing to do with any source list.

The limits of both readings

The citation figure comes from one bank of 630 questions in our own category, read on two apps on one date. The robots reading is 26 domains, and 24 of them were readable; two returned 403 and are reported as not usable rather than as blocks, because a 403 from one of them has already been shown in this programme to be a rate limiter. Two apps were read in the audit and our product covers five surfaces, so Perplexity, Claude and Google AI Overviews are unmeasured in the 197.

We sell the work of getting a business read correctly by these agents, which is the reason we publish our own access result too: the benchmark page returns the identical article to all eleven agents tested, at 63KB and 1,789 words. The search side of our own record sits on our proof page.

How unusual this policy is, counted

Rare, on the evidence of the sample. Across the 24 readable files in that 20 October reading, the counts of hosts blocking each agent at the root run like this.

Agent	Blocked at root, of 24	Named explicitly
GPTBot	0	9
OAI-SearchBot	0	7
OAI-AdsBot	0	0
PerplexityBot	1	9
ClaudeBot	1	9
Bytespider	6	9

Read the PerplexityBot and ClaudeBot rows against the finding above and the arithmetic is plain: the single host blocking each of those two agents is this publisher. Nobody else in a sample drawn from the most cited domains in our own audits, six named specialist competitors, the adjacent suites and directories, four small agencies and our own site does the same thing.

Questions about news publishers in AI citations

How many readings cited theverge.com?

197 of 1,260 on 16 October 2026, which makes it ninth in that top ten and the only news publisher in it.

Which agents does it allow?

At the root of its robots.txt as we read it on 20 October 2026, GPTBot and OAI-SearchBot are allowed and the other eleven of the fourteen we tested are blocked.

Does a robots.txt block explain the 403s?

The two findings corroborate each other for PerplexityBot and ClaudeBot, which are blocked in the file and refused at the edge. A file and an edge are separate layers, and elsewhere in the same sample they disagree.

Does being blocked stop a publication being cited?

This host was cited 197 times while refusing most of the list, so the two facts coexist. What the block decides is which agents can fetch the page now, and that is the part a placement strategy depends on.

How much of the citation pool do news sites hold?

3.2% of the 20,143 citations recorded on 16 October 2026, against 82.0% on vendor and agency own sites. The earlier run of the programme, with its own figures and method, is 597 readings across ChatGPT and Gemini in three markets.

One file, read in a minute, before the next invoice

The lesson here is cheap to apply. Whenever a placement is proposed, open that publication's robots.txt and see which agents are named. A title that permits two of fourteen is a different purchase from a title that permits all fourteen, at the same price, and the file says which one you are being sold. That check is part of the first week of work on every account we take, and you can run it yourself this afternoon.