Why your site audit reports no issues: the check that cannot fail

An audit that reports no issues is either a clean site or a check that read nothing, and the two outputs are identical. There is no error message, nothing throws, and the wrong answer looks exactly like the right one. The only defence that works is arithmetic: count how many things your check actually read, print that count beside the number you expected, and refuse to believe the result when they differ. We have five of these on record from a single fortnight of our own work, and every one of them produced reassurance from an empty input.

This is the most dangerous bug class in site auditing because its symptom is the result you were hoping for. A crash gets fixed in ten minutes. A check that silently passes everything can sit in a pipeline for weeks while people make decisions on it, and the longer it runs the more it is trusted.

The five, each one real and each one different

• A parameter accepted and ignored. A geographic targeting parameter was sent to an interface that returned 200 and did nothing with it. Every request succeeded. Every result described the wrong market.
• A classifier pattern that required a literal string. Our source classifier needed a particular text fragment to recognise official documentation, so an operator's own documentation was sorted as vendor marketing. The published figure for vendor own site citations read 91.3% until it was caught. The correct figure is 85.6%, and all classification now runs through one tested file that refuses to load if any of 24 named examples sorts wrongly.
• A threshold above the maximum. A similarity threshold was set higher than the highest score the data could produce, so nothing ever crossed it and the report was always empty.
• A word boundary eaten by a shell. A pattern written into a script through a shell lost one escape character and became a literal control character. No substitution ran at all and every similarity score came back comfortably safe. The same shape of bug hit a label filter the same morning, which reported 2 drops when the real number was 24.
• A guard testing the wrong property. A guard meant to catch deleted prose tested whether individual words still appeared somewhere on the page. Remove a whole sentence and every word in it still exists elsewhere, so no run of missing words ever formed and the guard waved the deletion through.

A sixth, from the same period, is the one that convinces people. A planner checked each candidate against a lookup table of patterns. One label was missing from the table, the lookup returned nothing, and the code treated nothing as no pattern to check against, so it kept the row. Fifty candidates that were already published went out as unwritten work. Nobody was told, because the only number printed was the count that got through.

The same shape outside code review

It is not only scripts. A style rule reading a colour token with a white fallback painted a set of statistic tiles white on a dark background, because the token it named did not exist. The page rendered. Nothing logged. A fallback that renders is not an error anywhere, which is the sentence worth carrying out of this whole subject.

And a verification pass once reported two claims confirmed on the basis that a particular string did not appear in a document. The document had not loaded. Absence of a string in a file you failed to read is evidence of nothing, and the fix was to make every claim name the document it depends on and mark any claim whose document failed to load as unverified rather than confirmed.

The four rules that catch all of them

Rule	What it stops
Count what you read and compare it to what you expected	A scan that fetched almost nothing and found no failures
Never let a missing lookup entry mean passed	A check that could not run reporting that it ran
Report what a filter dropped, not only what survived	Silent exclusions nobody sees for days
Plant a known positive and a known negative, and abort on either failure	A checker nobody has tried to fool

The fourth is the one people skip and the only one that generalises. A checker needs two self tests it runs before it does any work: something it must flag, and something it must not. Feed it a deliberate near copy of the thing it hunts and confirm it fires. Feed it two unrelated pieces of text and confirm it stays quiet. If either fails, stop the run rather than reading the output.

Measure your floor before you trust a threshold

A threshold is meaningless until you know what noise scores on your own data. We found this on a set of pages sharing a mandated title prefix: two deliberately unrelated subjects scored 0.607 against each other purely because of the shared words, which is above thresholds people routinely treat as a match. On that measure pure noise looked like duplication. Stripping the shared prefix and scoring only the distinct clause brought the same pages back to a real reading.

Two practical consequences. Compare the part that differs, not the part every item shares. And combine your measures by taking the lower of them rather than the higher, because a short sentence will collide with almost anything on one measure and taking the higher lets one accidental agreement win. A planted near copy still scores 0.714 under the stricter combiner, so nothing real is lost.

Count the blast radius before you act on a flag

The inverse failure is worth a line, because it is just as expensive. A flag is a lead and not a verdict. On one occasion a reported defect on two pages turned out, after a sweep of all 1,721 urls on the site, to touch 32. A fix applied on a wrong premise would have put the same falsehood on all 32 instead of correcting two. So read the sentence your check flagged, check it against the code or the product rather than against another page, size the blast radius, then fix the ones that survive.

Questions about silent failures in a site audit

My audit returned zero issues. How do I know it ran?

Print the number of items it read and compare it with the number you believe exist. If your sitemap has a thousand urls and your audit read nine hundred, you have a second problem before you have an answer to the first.

What is the single cheapest check to add?

A deliberate failure. Feed your audit one item you know is broken and confirm it reports it. If it does not, the clean report you have been reading means nothing, and you will know in under a minute.

Why write scripts with an editor rather than through a terminal?

Because a shell will consume escape characters inside a pattern and leave something that looks almost right. That exact failure hit us three times in two days, and in each case the result was a pass. Patterns belong in a file you can read back.

Should a check that cannot complete report a pass or a failure?

Neither. It should say it could not run and stop. Returning the answer that lets the work continue is how a missing table entry becomes fifty duplicated pages.

How do I know my similarity threshold is set sensibly?

Measure two things you know are unrelated and two you know are near copies, on your own data, before you set it. If noise scores near your threshold, the threshold is decorative.

One question, asked of every report you like the look of

What would this have told me if it had been handed nothing? Every failure above answers that question with the words you just read, and asking it out loud costs nothing. A published figure of ours spent a day wrong because nobody asked it, and the correction, the date and the tested replacement are all on the record: the figure that was wrong, the figure that is right and the scan behind both of them sit on our proof page alongside the exports and their windows. Selling a measurement means the bugs in the measuring tools are our problem rather than the reader's, which is why ours are listed by name here. What each separate reading on this site is allowed to answer, and the arithmetic that is not permitted between them, is set out in the four readings this site holds, and the evidence standard those readings are held to is in numbers and evidence required.