Eleven of twenty six hosts looked unreadable until we followed the redirect

The first pass of our robots.txt reading on 20 October 2026 came back with 15 of 26 files in hand and eleven hosts recorded as unreadable. The second pass, identical except that the client followed redirects, came back with 24 of 26. The eleven hosts had not changed, had not refused anything and had no policy about us at all. They answered the request for /robots.txt with a redirect, and a client that does not follow one treats that answer as nothing. Had the first numbers gone out, the published finding would have been about our own HTTP client rather than about anybody's crawler policy.

This is worth printing because the mistake is almost invisible from the inside. The first pass did not error. Nothing threw. It produced a tidy table with eleven blanks in it, and a blank in a policy table reads as reticence on the part of the host. The corrected reading, with the per agent counts across all 24 readable files, is published as 26 robots files with every rule counted.

What the two passes actually returned

Same 26 hosts, same minute of the same day, one setting different.

The two that stayed missing after redirect following are the only honest absences in the set. Both returned HTTP 403 to the request for the file itself, and both are reported as not usable rather than as blocks, because one of them has already produced a 403 in this programme that turned out to be a rate limiter and served the full article hours later. Nine of the eleven recovered hosts were simply on the other side of a hop.

The four hops a robots request runs into

None of these is unusual and none of them is a signal about artificial intelligence. They are the ordinary furniture of a site that has been alive for a few years.

• Bare domain to www, or the reverse. A host that canonicalises one way answers the other with a redirect, and the file lives only at the canonical name.
• HTTP to HTTPS. A request sent without the scheme forced lands on the plain port and gets moved.
• A trailing path normalisation. Some edges rewrite the path before the origin sees it.
• A regional or consent hop on hosts that route by country before serving anything.

Each one is a 301 or a 302 with a destination in the header. A browser follows it without telling you. A terminal client follows it only if you ask, and several common libraries default to not asking for exactly the reason that a silent hop can hide a change of host.

Why the wrong version of this table would have been the worse finding

A table of robots rules is read as a table of decisions. Eleven blanks in it invites a reader to conclude that eleven publishers have something to hide, or that eleven files are missing, or that a third of the category refuses automated requests. All three of those are claims about other people, built on a flag in our own code.

That shape has bitten this programme before and in the same direction every time. A classifier pattern that required a literal string counted an operator's own documentation as vendor marketing. A market parameter was accepted, returned success and did nothing. A verification script reported two claims confirmed while the document it was reading had failed to load. Every one of those produced a clean output and a wrong answer, and the general version of the problem is set out in why a site audit that reports no issues may be a check that cannot fail.

The rule we now run this check under

• Follow redirects, and record every hop. The chain is data. A robots file that lives at a different host from the one you asked about is a fact about the site, and you only have it if you wrote the hops down.
• Record the final URL beside the file. Under the Robots Exclusion Protocol the rules apply to the host, protocol and port serving the file, so the address you ended on decides what the file governs.
• Separate a missing file from an unobtained one. A 404 means the host publishes no file, which is itself a policy. A 403, a timeout or a hop you declined to follow means you do not know yet.
• Count both numbers in public. We fetched 26 and read 24. Printing the pair is what lets a reader see where the uncertainty sits.
• Re-read the response body on anything that is not a 200. A browser challenge asking for JavaScript is not a refusal, and it looks identical to one in a status column.

Flip one setting and fetch your list twice

This takes about ten minutes for a list of thirty hosts and it is worth doing before any conclusion goes in front of anybody.

• Fetch each host's file twice, once with redirect following off and once on, and diff the two result sets. Any host that appears only in the second set was a hop and never a policy.
• Print the status of every hop rather than the last one. A 200 at the end of a chain that passed through a 302 to a different hostname is a different reading from a 200 at the first address.
• Confirm the content type is plain text. An HTML page served in that slot is itself the finding, and it means no crawler is reading the rules you think you published.
• Keep the raw bytes of every response. The question of whether a 403 was a refusal or a challenge is answerable afterwards only if you saved the body.

Google's own documentation set is where the protocol questions get settled rather than argued, and which document answers which question is laid out in our guide to the documentation Google publishes itself.

Redirects, hops and unreadable files: what readers ask next

Why would a request for robots.txt be redirected at all?

Because the file sits at the root of a host, and most sites have more than one name for their root. A redirect from the bare domain to the www name, or from HTTP to HTTPS, catches the request before the file is served. The host is not doing anything unusual and a browser would never show you the hop.

How many of the eleven were genuine refusals?

None. Following redirects recovered nine of them outright and the remaining two were already accounted for in the second pass. The only hosts still unread after redirect following are the two that returned HTTP 403 to the request for the file, and those are published as not usable rather than as blocks.

Does the file at the end of a redirect govern the site I asked about?

It governs the host, protocol and port that served it. So the final address is the one to record, and where a hop lands on a different hostname that is a fact worth keeping in the row rather than flattening away.

Is a 404 on robots.txt the same as an unreadable file?

No, and the difference matters in a published count. A 404 is an answer: the host publishes no rules, so a well behaved crawler proceeds as though everything is permitted. An unobtained file is an open question, and putting the two in one column turns a measurement into an impression.

Which pass produced the published counts?

The second one, with redirects followed: 24 readable files of 26 fetched on 20 October 2026, parsed into the rule group that applies to each of fourteen named agents. The per agent totals and the per host list are both on the counted reading of those files.

The pass that gets thrown away is the one worth writing down

Nine hosts were nearly recorded as having withheld a public file they serve to anybody who asks properly. The gap between those two readings was one boolean, and the only thing that caught it was re running the fetch with the setting flipped before anybody wrote a sentence. We sell the measurement of whether assistants name a business, and the reason we can charge for it is that the arithmetic behind a number is checkable rather than asserted: our own Google Search Console exports, with their windows and the date each one was pulled, sit on our proof page, and the citation run that chose these 26 hosts is published as 597 readings across ChatGPT and Gemini in three markets. A reading we had to take twice is cheaper to publish than a reading we would have had to withdraw.