The most refused crawler in our sample is not an assistant at all
Bytespider is blocked at the root by 6 of the 24 readable robots files we read on 20 October 2026, more than twice any assistant agent in the same sample.
Published by AI Knows Us · Updated 19 October 2026
Reading the robots.txt of 26 domains our own citation audits had named, on 20 October 2026, we found 24 readable files and one agent refused far more often than the rest. Bytespider is blocked at the root by 6 of the 24. Every assistant agent in the same sample is blocked by one or two, and three of them by none. So the files do not say that sites are closing themselves to AI. They say that a handful of site owners have decided to refuse one bulk collector and have left the assistants open.
That distinction changes what the phrase blocking AI crawlers means, and it is checkable by anybody, because a robots.txt is a public file.
The refusal counts, in order
| Agent | Blocked at root, of 24 | Allowed at root |
|---|---|---|
| Bytespider | 6 | 18 |
| CCBot | 2 | 22 |
| Applebot-Extended | 2 | 22 |
| meta-externalagent | 2 | 22 |
| ChatGPT-User | 1 | 23 |
| ClaudeBot | 1 | 23 |
| Claude-User | 1 | 23 |
| Claude-SearchBot | 1 | 23 |
| Google-Extended | 1 | 23 |
| PerplexityBot | 1 | 23 |
| Perplexity-User | 1 | 23 |
| GPTBot | 0 | 24 |
| OAI-SearchBot | 0 | 24 |
| OAI-AdsBot | 0 | 24 |
Which hosts are doing the refusing
Five hosts in the sample refuse anything at all at the root, and one of them is responsible for most of the assistant blocks on its own.
- github.com blocks Bytespider.
- g2.com blocks Bytespider.
- capterra.com blocks Bytespider.
- rank4ai.co.uk blocks Bytespider.
- brightlocal.com blocks CCBot, Bytespider, Applebot-Extended and meta-externalagent.
- theverge.com blocks eleven of the fourteen and allows GPTBot and OAI-SearchBot.
Every other readable host in the sample blocked none of the fourteen. Take the news publisher out and the entire assistant facing block list in this sample disappears, which is why reading the per host view matters as much as the per agent totals.
What the pattern suggests site owners are actually deciding
Four of the six blocking hosts refuse exactly one agent, and it is the same one each time. That is not a position on artificial intelligence. It looks like a specific operational decision about one bulk collector, taken by sites whose content is their inventory: a code host, two software directories and a small agency.
Three consequences for anybody writing their own rules.
- Separate the decisions. Refusing a bulk collector settles how your writing may be reused. Refusing the fetcher that runs because one person asked a question settles whether you appear in that person's answer. Two different choices, and a single file can hold both.
- Do not copy a file you have not read. Copying a publisher's file copies a commercial position you may not hold, and in this sample the publisher's file is the outlier rather than the pattern.
- Write the date next to a named rule. Operators add identities, so a rule listing names is a restated document that goes stale without reporting anything.
The question of which crawler to allow, and how to measure the answer on your own logs rather than inheriting somebody else's, is worked through in a dated robots.txt and access control test.
Where this reading stops
Twenty six domains on one date, chosen because our citation audits of 15 and 16 October 2026 surfaced them, which makes this a reading about one category rather than about the web. Two of the 26 returned HTTP 403 and are reported as not usable rather than as blocks, since a 403 from one of those hosts has already turned out in this programme to be a rate limiter. The counts describe the root of each site and not deeper paths. A permission is also not an access grant: elsewhere in the same dataset a host allows all fourteen agents and refuses five of them at the edge, confirmed on two readings.
We sell the work of getting a business read and quoted correctly by these agents, so our own domain is in the sample and our own citation figure from the audits is on the record too: 6 readings of 1,260. The Google Search Console side of our own evidence, two domains with their windows and the export date printed, is our proof page.
What a sample like this is useful for, and what it is not
It is useful as a prior. Before diagnosing your own absence from answers in this category, it is worth knowing that among the sites already being cited, permission is almost never the thing being withheld. That moves the first hour of work from reading your own robots file to fetching your own pages under each agent name and reading what comes back.
It is not useful as a picture of the web, and it would be wrong to present it as one. Twenty six domains chosen because our audits named them is a sample of one category's cited set. A news archive, a research library or a stock photography site would sort differently, and the way to find out is to read the files for the domains that turn up in your own readings rather than ours.
Questions about which crawlers get blocked in robots.txt
Which agent is refused most often?
Bytespider, by 6 of the 24 readable files read on 20 October 2026. The next most refused are blocked by two each.
How many of these sites block assistant crawlers?
One, in the case of each of the named assistant agents, and that one is the news publisher in the sample. Three agents, GPTBot, OAI-SearchBot and OAI-AdsBot, are blocked by none of the 24.
Does this mean publishers are not blocking AI?
In this sample of 26 domains on this date, the only host turning assistant crawlers away in its file is the one news organisation. The other refusals are aimed at a bulk collector rather than at an assistant.
Should I block Bytespider too?
Decide what the rule is for before you copy it. Four of the hosts blocking it sell access to content or to listings, and a business whose pages are marketing is in a different position from a business whose pages are inventory.
Which audits chose these domains?
Our citation runs of 15 and 16 October 2026 on ChatGPT and Gemini, plus the separate city scan. The earlier run, with its full method and source mix, is published as 597 readings across ChatGPT and Gemini in three markets.
Read the per host column before you quote the per agent total
Six refusals of one agent sounds like a trend and turns out to be four sites making the same narrow operational call, plus one publisher with a commercial policy. That is the whole value of printing both views of the same dataset: the totals tell you how often something happens and the host list tells you who is doing it, and only the second one tells you whether it applies to you.