Why our own test passed: two requests that shared the same blind spot

Ask what your two test arms have in common, because that is where a passing test hides a fault. We already ran a check designed for exactly the problem we later found: it fetches a page as each named AI crawler and again as a browser, and it fails when a figure the browser can see is missing for the crawler. It never fired. Run from our own network, both arms received rupee figures, so there was no discrepancy to detect. The check was working correctly and it was structurally incapable of seeing the defect, because the thing that was wrong was wrong for both requests identically.

This is not a bug in the comparison. It is the nature of comparisons. A differential test is cheap, it needs no expected value and it catches a large class of real problems, and in exchange it inherits every assumption its two arms share. Whatever is true of both requests is invisible to it by construction.

What the two arms actually shared

Five things, and any one of them can hide a fault.

• The network. Both requests left from the same place, so both were treated as coming from the same country by anything reading a network signal.
• The absence of a prior session. Neither arm carried a cookie set on an earlier visit, so a signal stored from a previous session was missing from both.
• The moment. Both ran seconds apart against the same deployment, so anything that varies by build or by time of day was constant.
• The data behind the page. Both read the same records, so a wrong value in the source looks identical from both sides.
• The question being asked. Both arms asked whether the two responses differ, which is a weaker question than whether either response is correct.

The last one is the general case. A comparison answers whether two things agree. Agreement is not correctness, and two readings produced by the same pipeline agree far more often than they are right.

What a differential check can and cannot see

Fault	Caught by a crawler against browser comparison
A figure rendered only by a script, where the browser sees it and a fetcher does not	Yes. This is what it is for
A refusal served to a named agent and not to a browser	Yes
A value missing from the document for everybody	No. No difference exists
A region specific answer that is wrong for the region you did not test from	No. Both arms are in the same region
A wrong number in the underlying data	No. Both arms read it

Three ways to break a shared blind spot

• Vary one arm's environment on purpose. Run the same fetch from a second network, from a cloud server as well as an office connection, with no cookies and no language header, and from an address in another country if you can get one. Each variation removes one shared assumption.
• Replace the comparison with an assertion. Instead of asking whether two responses agree, state what the response must contain and require it. An assertion carries the expected answer inside the test, so it cannot be satisfied by two identical wrong readings.
• Add a third arm that resembles neither. A request with no agent string at all is the cheapest version of this and it finds things both named arms miss, because rules keyed on an expected header behave differently when the header is absent.

The second of those is usually the right trade. A comparison needs two requests and tells you about their relationship. An assertion needs one request and tells you about the thing you care about, which is whether your page says what it should. Who in an organisation owns a check like this, and at which stage it belongs, is set out in our workflow standard and its six stations.

The same shape, three more times

Once you recognise it, this pattern turns up everywhere in testing, and all three of these are from our own work in one fortnight.

A guard built to catch deleted prose tested whether individual words still appeared on the page. Remove an entire sentence and every word in it still exists elsewhere in the article, so no run of missing words ever formed and the deletion passed. The membership test and the defect shared an assumption about what loss looks like.

A verification pass reported two claims confirmed on the grounds that a particular string did not appear in a document. The document had failed to load. An empty input and a document that genuinely lacks the string produce the same result, and the check could not tell them apart.

And a disproof pass re-ran four disputed urls in reversed agent order inside the same window, 44 requests, all agreeing with the original. Running the same loop again reproduces whatever state the first run created, so the second pass and the first shared the condition that caused the result. The commands for running these fetches properly, including the arms that are worth adding, are in our guide to telling whether your website is readable by AI crawlers.

Questions about tests that pass when they should not

How do I tell whether a passing test means anything?

List what its inputs have in common and ask whether the fault you are worried about could live there. If it could, the pass carries no information about that fault and you need a different arm or a different kind of test.

Is a comparison test worth having at all?

Yes, for the class it covers, which is real and large. The mistake is treating it as coverage of a question it was never asking. Keep it and add an assertion beside it.

Why does the network matter so much?

Because a surprising amount of what a server decides about a request comes from where the request appears to come from, including which prices and which language a visitor is shown. Two arms from one network are two readings of one region.

What is the cheapest extra arm to add?

A request with no agent string, no cookies and no language header. It costs nothing, it takes the same path a crawler takes, and it finds the problems that a browser arm and a named agent arm agree about.

Should a test that cannot run report a pass?

It should report that it could not run and stop. Returning the answer that lets the work continue is how an unloaded document becomes two confirmed claims, and it is the same failure in a different coat.

Agreement is not correctness

The sentence worth keeping from all of this is that two requests agreeing proves they were treated the same way, and nothing more. Our own gate agreed with itself for weeks about a page that was telling an American buyer the wrong price, and the only thing that found it was somebody asking what both arms had in common. We publish the gates that failed us alongside the readings they were supposed to protect, with dates on both, and the exports, windows and dated blind scan behind every figure we quote are on our proof page. Running this kind of check on other companies' sites, and replacing the ones that cannot fail, is the work we charge for.