When a finding would be the worst possible news, that is the moment to try hardest to disprove it

Before you report an alarming result, list at least two other things that would produce the same evidence. We did not, once, and it cost an hour on the single worst finding available about a company whose product is being read correctly by machines: a report that no page body on this site was served to a crawler. The evidence was genuine. A fetch had returned a document with the navigation in it and no article. The explanation was that the url did not exist, because it had been assembled from a slug rather than read from the sitemap. The article is served identically to eleven agent strings at HTTP 200, 63KB uncompressed and 1,789 words.

The uncomfortable part is that the finding was not careless. The check was the right check, the person running it was looking for exactly the right class of problem, and the response that came back really did look like a rendering failure. What was missing was a single question asked before writing it down.

Why the worst news gets the least scrutiny

You would expect the opposite, and the mechanism is worth naming because it is situational rather than personal.

• Urgency feels like responsibility. If something this serious is true, every minute of delay looks like negligence, so the incentive runs towards reporting immediately.
• A dramatic result is socially rewarded. Nobody is congratulated for the reading where everything was fine, which quietly biases how hard each kind of result gets tested.
• The evidence is real. This is the trap. The response really was empty. What was wrong was the inference, and inferences are much harder to doubt than data.
• The alternative explanation is boring. A wrong url is a dull answer to an exciting question, and dull answers get considered last.

The four questions, in order

These take about two minutes between them and they would have caught every false alarm this programme has produced.

• What else would produce exactly this evidence? Write down two alternatives before anything else. For an empty response the list is short and obvious once you are forced to make it: the page does not exist, the request was refused, the page exists and is rendered by a script.
• What is the cheapest reading that separates them? Here it was the status code, which was already in the response, plus one fetch of a url known to be good. Neither required any new tooling.
• What would this check have reported if it had been handed nothing? If the answer is the same thing it just reported, the result carries no information. This one question has caught five separate bugs of ours in a fortnight.
• Who else can take this reading, later, from somewhere else? A reading taken once by one party is a lead. The second one, deliberately spaced and ideally from another network, is what makes it a finding.

Two more times the discipline paid, and what it saved

In an access run of 176 attempts across 16 public urls under 11 agent names, 23 came back refused, and we reported four hosts as answering differently depending on the agent name. A later reading from another network returned 200 and the full article to almost every agent on two of them, with an explicit HTTP 429 from one host. They were rate limiters. Two of four findings were withdrawn, and the withdrawal is published with the same prominence as the rows that held, in our bot access audit where only two of four survived.

Separately, two careful readers reported five defects on live pages between them. One was real and was fixed. Four were not, and three of those turned on the same thing: a sentence scoped by its own wording. A claim written as what this page publishes, or as what was true on a stated date, does not become false when later work goes beyond it, however much it reads like a present tense claim at a glance. Acting on those four would have put the same falsehood on 32 pages instead of correcting two, because a sweep of the whole site showed the sentence pattern was far more widespread than the reports suggested.

The asymmetry that pays for the extra minutes

Scenario	What it costs
A true finding reported ten minutes late	Ten minutes
A false finding published	The credibility of every other number you publish
A false finding acted on	Correct work undone across every page the fix touches

That is the whole argument. The downside of pausing is bounded and small; the downside of publishing a confident error is unbounded, because a reader who catches one invented or mistaken figure is entitled to assume the rest were produced the same way.

What to publish when you cannot settle it

Sometimes two readings disagree and no third reading is available. The honest output then is not silence and it is not the more interesting of the two readings. It is a statement of what you measured, what you observed, and which question you are not able to answer, with the dates on all three. Two of our access hosts sit in exactly that state and are described as hosts we will not characterise, which is a less satisfying sentence than a table row and a far more defensible one. A pair of readings that genuinely disagreed, published as a pair rather than resolved into one, is set out in our page on one question won and one question lost on the same day.

Questions about handling an alarming result from your own audit

How long should I hold a serious finding before reporting it?

Long enough to run the cheapest reading that would disprove it, which is usually minutes rather than hours. If the disproof is expensive, report the observation and say explicitly that it is one reading.

What if pausing means a real problem stays live longer?

Report the observation to the people who can act, and label it as unconfirmed. The thing to delay is the conclusion, not the conversation, and those two get collapsed together far too easily.

How do I avoid talking myself out of a true finding?

By making the test symmetrical. Write down what evidence would confirm it as well as what would refute it, then go and get both. A disproof attempt that is designed to fail is as useless as no attempt.

Does this apply to findings about other companies?

More so, because you cannot see their configuration and the cost of being wrong lands on somebody who did not choose to be measured. Every named host in our own published readings has at least two readings behind it.

Should the correction be as prominent as the original claim?

Yes, and in practice the correction is the more interesting page. Across 102 conversations about what makes a page in this category worth citing, 68 asked for failures and negative results to be shown, and almost nobody in this market publishes them.

The hour bought a rule

The sixty minutes spent on a false alarm produced the check that every reading in this programme now starts with, which is to take the url from the sitemap and read the status before concluding anything. That is a reasonable return, and it is only available if you write the mistake down rather than quietly deleting it. Ours are kept next to the results we like, with the dates on both, and the exports, windows and the dated blind scan behind every published figure sit on our proof page. We are paid to take these readings on other companies' domains, which is precisely why the ones we got wrong about our own are on the record.