Four small agencies the engines cited, and between them they block one crawler

Our own citation runs surfaced four small agencies that kept turning up in answers, and on 20 October 2026 we read the robots.txt of all four. Three of them, whito.co.uk, aeoscore.co.uk and themediaverse.in, block none of the fourteen named agents we tested at the root. The fourth, rank4ai.co.uk, blocks exactly one, and it is Bytespider, a bulk collector rather than an assistant. So the sites that were winning in our data have permission files that say yes to every agent capable of fetching a page to answer a live question. Whatever these four are doing to get named, closing or opening the door is not it, because the door was already open and so is almost everybody else's.

That is a more useful finding than it looks, because it removes the cheapest available explanation. Across the whole 26 host sample not one of the 24 readable files refuses GPTBot, OAI-SearchBot or OAI-AdsBot at the root. The winners and the rest of the field share the same configuration, so the difference that produced the citations has to be in the pages. The citation counts themselves, market by market, are published in the United Kingdom cut of 204 readings.

The four files, read at the root

The fourteen agents tested were GPTBot, OAI-SearchBot, OAI-AdsBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Applebot-Extended and meta-externalagent. Three of these four files name none of them, which means every agent falls into whatever the wildcard group says and the wildcard group is open.

What a file like this actually signals

A small agency's robots.txt is usually short, and the shortness is the decision rather than the absence of one. Three things follow from these four files.

• Nothing here needs protecting from reuse. These sites publish marketing. A marketing page being read widely is the point of writing it, which is a different commercial position from a directory selling access to listings or a publisher licensing an archive.
• A short file has fewer ways to go wrong. The commonest robots defect in this area is a named group that accidentally reopens something the wildcard group was closing, because under the Robots Exclusion Protocol the most specific matching group wins and the wildcard stops applying to that agent. A file that names nobody cannot make that mistake.
• One named rule is a maintained rule. rank4ai.co.uk naming a single bulk collector is somebody who read an operator list and made one decision, which is the shape four other hosts in the wider sample arrived at too.

So what did produce the citations

Our audits recorded these four as winning on specificity rather than on size. The pattern across the local readings was consistent: the hosts that got named were the ones whose domain and whose pages were about one named place or one named job, and the same shape held in the ten city scan where every city's most cited local source was a small, city specific domain.

• The place or the specialism is in the page, not implied by it. A page about a city names the city in the title, the first paragraph and the body, because that is the string a question contains.
• The claim is narrow enough to be checkable. A specific service, a specific market, a specific price band. A general promise has nothing in it for a machine to quote back.
• The facts are in the first response. If a figure arrives only after a script runs, an agent that fetches the document and reads it has not got the figure. Our own pricing page failed exactly that test on 16 October 2026 and the fix was to put every currency in the HTML.
• The label on the site matters less than what the pages do. Working out what a supplier's own description actually commits it to is a separate skill, and the version of it for one common label is set out in what the inbound marketing agency label means in practice.

Reading a competitor's file to the same standard

These four rows took about two minutes each and anybody can repeat them, because a robots.txt is served to everyone.

• Fetch the file with redirect following on. Our own first pass without it lost eleven of twenty six hosts, and a blank in a policy table reads as a refusal when it was a hop.
• Parse it per agent rather than reading it with your eyes, and record which group each agent matched, named or wildcard.
• Write all fourteen names down with allowed or blocked beside each and the date, because these are files their owners can edit this afternoon.
• Then fetch a real page under each agent name, because a file and an edge are separate layers and in this same sample a help centre permits all fourteen agents and refuses five of them at the edge on two separate readings.

What four files on one date will and will not carry

Four files on one date, read at the root, and the counts say nothing about deeper paths. The citation counts beside them come from a bank of 102 questions per market read on ChatGPT and Gemini on 15 October 2026, which is two engines rather than the five surfaces our product watches, and they count a domain appearing in a source list rather than a business being named in an answer's text. Those two findings have different denominators and are never added. There is no per industry rate anywhere in our data, so nothing here supports a sentence about agencies in general.

Small agency files and citation counts: the common questions

Do the agencies winning AI citations block crawlers?

Three of the four we read block none of the fourteen agents we tested at the root, and the fourth blocks Bytespider alone. Read on 20 October 2026, and every file is public so the rows can be rechecked today.

Does leaving everything open explain why they were cited?

No, because almost every host in the wider sample is open too. Across 24 readable files, zero refuse GPTBot, OAI-SearchBot or OAI-AdsBot at the root. An open file is the normal condition in this category, which means it cannot be the difference between being named and being absent.

Should a small agency block the training crawlers?

Decide what the rule is for before copying one. A site whose pages are marketing gains from being read widely. A site whose pages are inventory, like a listings business or a paid archive, is in a different position, and in our sample it is exactly those businesses that named a bulk collector.

Why was one agency's file different from the other three?

It names one bulk collector and nothing else, which is a single maintained decision rather than a policy about assistants. Four other hosts in the sample made the same call, and the per agent and per host totals are in our counted reading of 26 robots files.

Which runs named these four agencies?

The citation run of 15 October 2026 across the United States, the United Kingdom and India on ChatGPT and Gemini, 597 usable readings, published in full as 597 readings across ChatGPT and Gemini in three markets, with the per market cuts published separately.

Four open files, and the thing they have in common is not the file

If you were hoping the answer was a configuration, these four rows close that off. The agencies our engines kept naming run permission files that are either empty of names or hold one line, which is the same file most of their competitors are running. What separated them was that their pages were about a named place and a named job in words a question actually contains. Writing those pages, and measuring which answers change afterwards, is what we charge for, and the standard we hold our own claims to is on our proof page, where each figure carries its window and the day the file behind it was pulled.