Has anybody measured whether AI visibility tools report accurately, and what that test needs

Every tool in this category reports a number. The question nobody in it answers is whether the number matches the answer it was taken from. We do not answer it either, and this page says what the test would be and why the absence is visible in our own data.

The evidence that the study has not been done

On 22 October 2026 we asked ChatGPT, Gemini and Claude what a vendor page in this category would have to publish to be worth citing, and kept 224 specifications. Three of them specified a platform comparison and each one demanded a measurement accuracy figure. Each one then supplied its own value.

• One asked for a comparative measurement accuracy score per tool with its sample size and annotation method, and added, in its own words, that if accuracy cannot be validated the ranking must not be presented as objective.
• One asked for the percentage of manually adjudicated observations in which a platform classification matches reference labels, with the exact numerator, denominator, date and engine mix.
• One supplied a headline of 98.4 per cent citation extraction fidelity across 10,000 multi engine prompt outputs tested on 1 October 2026.

A model asked for a convincing page produces a figure of the shape the reader expects. Three specifications asking for the same quantity and arriving at three unrelated treatments of it is what that looks like when the quantity has never been published. We also hold the citation side of the same run: 19,793 citation URLs across 3,307 hosts, and no adjudicated accuracy study appears among the sources those answers were built from. The host table is in the citation host study.

What the test would have to be

Taken from the specifications themselves, because the engines were specific about it, and reordered into the sequence a buyer could actually run.

1. Collect the ground truth by hand. At least 100 prompts per engine, with three fresh runs per prompt where practical, and the full answer text and sources block saved for each reading.
2. Label it twice. Two independent human annotators per answer for mentions and for citations, with the agreement rate published and a written rule for resolving disagreements.
3. Report citation precision. Confirmed citations divided by all citations the tool detected, with the number of underlying answers and the dates.
4. Report citation recall separately. The proportion of manually verified citations the tool found. A tool can win one of these and lose the other badly.
5. Report three URL match rates as three numbers. Exact match, canonicalised match and domain level match. Collapsing them into one hides the most common failure, which is a tool crediting a domain for a page it never cited.
6. Keep recommendation position apart from citation position, and assign no rank to a brand that was not mentioned.
7. Publish a dated sample of the raw answers and citation URLs so a reader can check the tool report against the original output.

That is between 400 and 1,200 hand labelled readings per vendor before a single accuracy figure exists. Nobody has published it, which is a statement about cost rather than about anybody's honesty.

Why repeats are not optional in such a test

On the same morning, 22 October 2026, we sent 22 buying questions to ChatGPT and Gemini three times each inside six minutes. A brand that appeared at all appeared in all three runs 53.0 per cent of the time on ChatGPT, 35 of 66 brand and prompt pairs, and 26.1 per cent on Gemini, 30 of 115. On Gemini, 53 of 115 pairs appeared in exactly one of three runs.

An accuracy test built on one reading per prompt therefore cannot separate a tool missing a mention from the engine not producing that mention on the run the tool happened to take. Three runs is the floor, and it is the reason an accuracy benchmark in this category costs what it costs. The full table is in the run to run measurement.

What the vendors publish instead, read on 22 October 2026

Limits, not accuracy. These are our own fetches on that date, each attached to the URL it was read on rather than to the company.

• tryprofound.com/pricing. No price figure in any currency anywhere on the page. Trial publishes 50 unique prompts run daily for 7 days across three answer engines named as ChatGPT, Gemini and Google AI Overviews, one language, one region, five prompt volume searches, no data history, no exports, no API, single sign on listed as No. Enterprise publishes up to 9 answer engines, all time history, CSV and JSON export, an API, unlimited seats, and SSO and SAML with SOC 2 compliance, priced as Custom. Agency Growth is listed with 400 credits a month per client workspace and no price. Any dollar figure quoted for this vendor did not come from this page.
• brightlocal.com/pricing. Two readings of one document, and both render. The plan card reads 31 dollars a month on the annual option with 369 billed annually. The per location table in the same page reads 41 dollars a month billed monthly for a single location, with the same 369 annual total. We publish both rather than the lower one.
• Our own pricing page. No accuracy figure, and no SSO, SAML, SOC 2, ISO, DPA or white label, checked by word boundary over 119,203 bytes. Growth at 299 dollars is the cheapest of our plans that reads all five surfaces, SEO Start at 149 renders 0 questions and 0 credits and 0 engines, and outside the United States and the United Kingdom SEO Start renders 99 and Growth 199.

Limits are the better thing to compare anyway. A prompt allowance, a run frequency and an export right are stable enough to hold a vendor to, and a price table is wrong within a quarter.

What to do while the benchmark does not exist

1. Take ten of your own buyer questions and ask them by hand in the engines, three runs each, on the same morning.
2. Ask your tool for the same questions on the same day.
3. Compare names, not scores. Count where the tool found a mention you cannot see and where you can see one it missed.
4. Write both counts down with the date. That is your own precision and recall sample, on your own category, and it is worth more to you than any vendor league table.

The grading criteria to apply to the tool afterwards are in how to grade an AI monitoring tool, and the reporting shape for the result is in how to report it to leadership.

Our own position, stated

We sell a tool in this category and we publish no accuracy benchmark for it. We also publish our own worst outcome number: across 635 readings on 22 October 2026 our company was named 0 times and our domain appeared in 0 of 19,793 citation URLs, after we had published 507 more pages in two days. That is in the 507 page null result. A vendor without an accuracy study is the normal case here, and the right response is to run the ten question test above on whoever you are considering, including us.

Questions about AI visibility tool accuracy

Does any vendor publish an accuracy figure

None that we read on 22 October 2026 on the pricing pages above. What they publish is limits, which is a different and more durable thing.

What is citation precision

Confirmed citations divided by all citations the tool reported, over a stated number of underlying answers, on stated dates. Recall is the other direction and both are needed.

Can I test a tool myself without 100 prompts per engine

Yes, at lower precision. Ten questions with three runs each gives you a sample you can act on, and it will catch a tool crediting a domain for a page it never cited.

Why does a tool report a mention I cannot reproduce

Often because the engine did not produce it on your run. On Gemini, 53 of 115 brand and prompt pairs in our test appeared in exactly one of three runs.

Until that benchmark exists, the protection is contractual, and the clauses are in what to put in a contract for AI search visibility work, clause by clause. Whether to pay for any of this, measured against our own null result, is in is generative engine optimisation worth paying for, and what our own null result says about it.