How to tell a rate limiter from a refusal, after it fooled us twice in five days

One explicit HTTP 429 anywhere in a run is the host telling you what the rest of its refusals were, and the correct response is to doubt the whole run rather than that single row. We learned it the expensive way. In an access run of 176 attempts, 16 public urls requested under 11 agent names, 23 came back refused outright, and we reported four hosts as answering differently by agent. One of those hosts then returned a plain 429 on a later reading. Two of the four findings were withdrawn. Five days later, in a completely separate run, the same trap was waiting in a different shape, and this time we recognised it.

This matters because the two causes need opposite responses. A policy refusal is somebody's decision and you change it by talking to them or by changing your rules. A limiter is arithmetic, and you change it by slowing down. Recording one as the other puts a false claim about a named company into a published table, which is a worse outcome than having no table.

Which count you are reading, because there are two

The access run is on record in two shapes and they are not interchangeable. 176 attempts with 23 refused is the first pass: 16 urls under 11 agent names. 220 dated fetches is that same pass plus the later confirming reading taken hours afterwards. Every figure in this section is from the first pass unless it says otherwise, and the habit of naming which pass a number came from is cheap to form and impossible to retrofit.

The second time it happened, and why it cost nothing

On 20 October 2026 we fetched the robots file from 26 domains our own audits had named. Twenty four were readable. Two returned HTTP 403 to a plain request for a public text file, and they are recorded as not usable rather than as blocks. One of those two is the same host that produced the withdrawn finding in the access run.

Nothing in that second run needed a retraction, because the rule written after the first one was applied before anything was published. The count in that study is 24 and not 26 for exactly this reason, and the smaller number is the defensible one. That is the whole return on writing a rule down.

The five signals that separate the two

• A 429 anywhere in the set. Not just on the url you care about. A limiter that answers one agent with a 429 is perfectly capable of answering the next one with a 403, and the status code it chooses is a configuration detail rather than a message.
• The response body. A plain forbidden page is a different artefact from a page that says it is checking your browser or asks you to enable JavaScript. In the access run, 20 of the 23 refusals carried a challenge body rather than a refusal, and that was sitting in the saved responses from the first minute.
• Whether a desktop browser string is refused too. A rule aimed at an agent name lets an ordinary browser through. A limiter counting requests from one address does not care what name arrives.
• A reading taken in a separate session, hours later, from another network. This is the one that actually settles it, and it is the only one that cannot be faked by running the same loop again.
• Whether the refusal tracks your volume or the name you sent. Request one url once under one name after a long gap. If it answers, you were being counted, not judged.

The two shapes, side by side

What you see	A policy refusal	A limiter
Status	Stable across sessions	Moves between 403, 429 and 503
Body	A short forbidden page	Often a browser check or a script requirement
A browser string	Usually served	Refused alongside everything else
Hours later, another network	The same answer	The full page, often with no refusal at all
One request after a long gap	Still refused	Served

Why a same day re-run does not help

Our first instinct was the right shape and the wrong execution. We re-ran the four disputed urls the same day in reversed agent order with gaps between requests, 44 requests in all, and every status agreed with the original. That looked like confirmation. It was the limiter still counting. Forty four requests to four hosts inside one window reproduce whatever state the first pass created, which is the opposite of testing for it.

So the third reading has to be deliberately spaced, and ideally taken from somewhere else. The full protocol and the two rows we removed are published in our bot access audit where only two of four findings survived, including the two hosts that did not make the final table.

What to do when you hit one on your own site

• Read the Retry-After header if the response carries one, and honour it. It is the host telling you the number you were missing.
• Put a real gap between requests and a long gap between agent names. Two seconds is not a gap to a counter that works in bursts.
• Save every body to disk, not just the status. The classification you need is in the body, and re-running a limited set is the one action certain to produce a different answer.
• Keep refusals, challenges and throttles in three separate columns in your own logs. They move independently, and one combined failure count hides which of them changed.
• Decide the question before you start. If you are asking whether a host refuses a named agent, you need one request per name, spaced, over days, and not a fast sweep.

Questions about rate limits during a crawler access test

Does a 429 mean I have been blocked?

It means you asked for more than that host wanted to serve in the time you took. It is a statement about rate and not about identity, and the usual fix is to go slower and come back. Treat it as an instruction rather than a verdict.

Can a limiter really return a 403?

Yes, and that is the whole reason this page exists. The status a bot management product returns when a threshold is crossed is whatever its configuration says, so one host in a run can answer with 429 while its neighbour answers with 403 for the identical reason.

How long should I wait before the confirming reading?

Hours rather than minutes, and from a different network if you can get one. A home or office connection and a cloud server are treated differently by bot managers, and that difference is useful to you rather than a nuisance.

What if the two readings disagree?

Then you have a host you cannot characterise yet, and saying so is the honest output. Two of our four candidate rows ended up in that state and we published them as removed with the reason attached, which is a more useful page than four confident rows would have been.

Does any of this change what an assistant says about me?

Not directly, and the two records belong apart. Access is about whether a page can be fetched. Being named in an answer is a separate measurement on separate dates, and our own figures for it are 3 citations of our domain across 597 readings of ChatGPT and Gemini on 15 October 2026 and 6 across 1,260 readings of those two engines the next day.

The discipline is the deliverable

Anybody can run a sweep and publish a table. The work that distinguishes a reading worth trusting is the part where you go back, read the bodies you already have, and take the number down. We hold ourselves to it in public: the exports, the windows and the dated blind scan behind every claim we make are laid out on our proof page, and the two withdrawn rows are published beside the two that held. Companies pay us to run this kind of measurement on their own domains and to fix what it turns up, so publishing the half that embarrassed us is the cheapest proof available that the method is real. If you are keeping your own record of these readings, the argument for storing the raw capture rather than your conclusion about it is made in our page on keeping the log rather than the shortlist.