Nothing in our pipeline noticed that two of its own figures disagreed. A reader did.

The thing that catches a wrong published figure is almost always a person holding two numbers and asking why they differ. An automated check tests a property somebody already suspected, it sees one output at a time, and it reports a status rather than a body. None of those three habits can produce the sentence that matters here, which is that two results from the same pipeline do not agree. Four corrections in our own programme, inside a single fortnight, were found by reading, and the four are worth naming because the cost of each would have been a published figure that stayed wrong.

This is uncomfortable for a company that sells measurement, which is the reason to write it down. We are paid to take these readings and to publish them, so the failures of our own quality process are ours to print rather than the reader's to discover.

The four, and what no check could have done

What was found	How it was found	What a check could not do
One share existing as two figures	Somebody compared them and asked which was right	Hold both outputs at once
Two cited hostnames that were not publishers	Somebody opened the cited addresses	Open an address it was counting
Refusals that were rate limiting	Somebody read the stored response bodies	Read a body when it was told to record a status
A pattern in one market and in neither of the others	Somebody read the output and noticed	Look for a pattern nobody had named

One: a share with two values

The vendor and agency own site share of our first citation run existed as 91.3% and as 89.1% before the correct figure, 85.6%, was established. The published correction names both of the superseded values for exactly that reason. Nothing in the pipeline held the two together, so nothing in the pipeline could notice. A person reading two documents could, and did.

Underneath it sat a sorting rule blind to a whole class of documentation host, so an operator's own pages were filed as marketing. There is now one place where that sorting happens, and it holds 24 worked cases as a precondition on its own startup. Why the older figure stays in print next to the newer one, with its cause attached, is set out in keeping the superseded number in print.

Two: two hostnames that looked like a finding

Two unfamiliar hostnames appeared in our first run, cited in 75 readings and in 88, both serving what looked like mirrors of Google's developer documentation. Read as rows in a table they were the most interesting result in the dataset. Opening the cited addresses settled it in under a minute: one is an iframe proxy and the other a reverse proxy mirror, both handing over the real documentation page under a name of their own, and both are an artefact of how the data was fetched rather than anything an engine did.

No counting step opens an address. A table of hostnames is exactly as confident about a proxy as about a publisher, and the only way to tell is to go and look. Once each address was pointed at the document it hands over, one run's share read 85.3% in place of 85.6%, and the other read 82.0% in place of 82.4%.

Three: the refusals that were about traffic

An access run of 176 attempts, 16 public addresses under 11 agent names, recorded 23 refused outright. That reads as a finding about which hosts turn away which crawlers, and part of it was. The part that was not is that 20 of those 23 responses carried a browser challenge body where a refusal should have been, and one host later answered with an explicit rate limit status.

Every one of those bodies had been saved. The information was inside the dataset from the first minute, and the run's own summary could not surface it, because the summary was a count of statuses and a challenge page returns the same status as a refusal. Somebody read the bodies.

Four: the pattern nobody had asked for

In the United Kingdom readings of the first run, gov.uk was cited in 57 readings and ico.org.uk in 28, and neither appeared in the United States set at all. That is a genuine finding about how one market gets answered, and it arrived because a writer read the output and noticed a shape, then reported it. No check was looking for it, because nobody had thought to name it before it existed.

It also exposed something about our own process rather than the data: the finding could not be cited at first, because the evidence file it had to be quoted from was narrower than the work. The file was widened. The writer was right to leave a real result out rather than guess at it, and the correct repair was to the file.

A report is a lead, which is the other half of this

None of the above argues for acting on everything a reader reports. On one occasion five defects were reported about live pages of ours and one of them was real. Acting on the other four would have put the same false sentence on dozens of pages, because a sweep showed the affected set was far larger than the two pages named in the report.

So the discipline runs both ways. Keep asking people to read the output, and keep checking each report against the code, the product or the statute before changing anything. The cost of the check is minutes and the cost of skipping it is measured in pages. Reviewing machine produced output before it ships is a general version of the same habit, worked through for one kind of generated text in what those generators actually produce and the review step that follows.

Where a gate is better than a person

Gates and readers are not alternatives. A gate is better at anything repetitive and already named: the module described above catches that first failure every time, forever, in milliseconds, and no person is going to match it. What a gate cannot be is curious. Hand it the cases you already know about, and spend the review hour on the two questions nothing automated will think to ask, which are why these two figures differ and what this address actually serves.

We hold no measurement of how often review catches what a gate missed, and four cases in a fortnight is four cases rather than a rate. The runs behind them were taken on two products. The other three we cover for a customer were in neither one, which is a fact about the product and not a description of these datasets.

Questions about reading the output of your own measurement

Which review question earns its hour?

Find two numbers in your own documents that should be identical and ask why they are not. That question produced the largest correction in our programme, and it needs no tooling at all.

How do I review a table of cited domains?

Click the top few entries and see where they land. A host that hands over somebody else's document is not a publisher, and discovering one that way costs less time than arguing about whether the table was assembled carefully.

Should every response body really be stored?

The access run settles it. A status code could not tell a refusal from a challenge page, and because the bodies had been kept the question was answerable later without sending a single request again.

Does this mean automated checks are not worth building?

They are worth building for the failures you have already seen. Every correction above is now a named example inside something that refuses to run when it recurs. The argument is about where the next unknown failure gets caught, and that has consistently been a person reading the output.

Who should do the reading?

Somebody who has to use the numbers in a sentence. Three of the four above were found by writers working from the data rather than by the people who built the pipeline, because writing a figure into a claim forces a look at where it came from.

The review that pays for itself is the one that asks a question nobody wrote down

Every gate in this programme exists because a person found something first, which makes the reading the primary control and the gate the record of it. Two numbers that disagree, an address that resolves somewhere unexpected, a stored body that says something different from its status code: those are the three places to spend an hour, and none of them can be scheduled by a tool that does not yet know what it is looking for. Our fee covers the reading and the pages that come out of it, so a list of our own missed checks is one of the costs of doing this in public. Our own list of them is kept on aiknowsus.com/proof, underneath the exports.