Why two overlapping ranges mean nothing has moved
A plain guide to telling a real change in AI visibility from ordinary sampling noise, with the arithmetic you can redo yourself.
Published by AI Knows Us · Updated 9 November 2026
Short answer: You know your AI visibility improved when the new mention rate and the old one have confidence ranges that do not overlap. A rate measured from a few dozen questions is an estimate, not a fact. 10 mentions out of 40 questions is 25 percent, but the honest range around it runs from about 14 percent to 40 percent. If next month's range starts inside last month's range, the sensible reading is that nothing has moved yet. This page is published by AI Knows Us and explains the rule we apply to our own numbers.
The problem: a percentage that looks like progress
You run a check in October. Your brand is named in 25 percent of AI answers. You publish some pages. You run the check again in November. Now it is 35 percent. Ten points. A slide gets made. A budget gets renewed.
The trouble is that both numbers came from a sample of questions, not from every question a buyer could ask. Ask 40 questions and you get one answer. Ask a different 40 the same afternoon and you get another. The underlying truth did not change between the two runs. Only the draw did.
This is the oldest problem in survey work, and it has a standard fix: report the uncertainty with the number. A mention rate without a range is a guess wearing a suit.
Why it matters more than it sounds
Three things make AI visibility measurement noisier than most dashboards admit.
- Small samples. Most tracking runs ask tens of questions, not thousands. At 40 questions, each single mention moves the headline rate by 2.5 points.
- Model variation. The same question asked twice on the same day can return different sources and different named brands. The answer is generated, not looked up from a fixed index.
- Engine disagreement. ChatGPT, Gemini, Perplexity, Claude and Google AI Overviews answer the same question from different sources. A rate blended across them hides which one moved.
Put those together and a ten point swing is often just weather. If you act on weather, you rewrite pages that were already working, or you keep paying for work that did nothing.
The rule in one line
Compute a 95 percent interval around each rate. If the two intervals overlap, report it as no movement. If they clear each other, report it as movement. That is a conservative rule, and we say so below.
Last verified: 08 November 2026
A worked example you can repeat
Here is the full arithmetic for a 40 question run. The interval used is the Wilson score interval, which behaves better than the simple textbook formula when the count is small or the rate is near 0 or 100 percent. The formula is:
( p + z²/2n ± z × √( p(1-p)/n + z²/4n² ) ) ÷ ( 1 + z²/n ), where p is the observed rate, n is the number of questions, and z is 1.96 for a 95 percent interval.
Worked for 10 mentions out of 40 questions (p = 0.25, n = 40):
- z² = 3.8416. z²/n = 0.09604. z²/2n = 0.04802.
- Centre = (0.25 + 0.04802) ÷ 1.09604 = 0.2719.
- Inside the root: (0.25 × 0.75) ÷ 40 = 0.0046875, plus 3.8416 ÷ 6400 = 0.00060. Total 0.0052878. Root = 0.07272.
- Half width = (1.96 ÷ 1.09604) × 0.07272 = 0.1300.
- Interval = 0.1419 to 0.4019, that is 14 percent to 40 percent.
Now the November reading, 14 mentions out of 40 (p = 0.35). Run the same five steps and you get 22 percent to 50 percent. The two ranges share the band from 22 to 40 percent. They overlap. Under the rule, nothing has moved.
Wilson 95 percent intervals at 40 questions
| Mentions | Rate | 95 percent range |
|---|---|---|
| 4 of 40 | 10 percent | 4 to 23 percent |
| 8 of 40 | 20 percent | 11 to 35 percent |
| 10 of 40 | 25 percent | 14 to 40 percent |
| 14 of 40 | 35 percent | 22 to 50 percent |
| 20 of 40 | 50 percent | 35 to 65 percent |
| 24 of 40 | 60 percent | 45 to 74 percent |
Read down the table and the point lands. Starting from 10 of 40, you need to reach about 24 of 40 before the ranges clear each other at this sample size. 22 of 40 (55 percent) still overlaps, just. That is a large real gain, not a nudge.
The other lever: ask more questions
Sample size shrinks the range. At 40 questions, 25 percent carries a range of 14 to 40. At 100 questions, the same 25 percent carries a range of about 18 to 34. Same rate, narrower claim. If you want to detect smaller changes, you buy that with more questions, not with better charting.
What we do about it
We apply the rule to our own output, including when it is inconvenient.
- Every rate is printed with a 95 percent range. Two numbers whose ranges overlap are reported as not having moved.
- Every tracking question is asked without naming the business. Naming it primes the engine and the reading becomes worthless, so a primed number is not comparable to a blind one.
- Readings are taken across ChatGPT, Gemini, Perplexity, Claude and Google AI Overviews, because they answer the same question differently on the same day from different sources.
The uncomfortable part of this is commercial. A measurement method that refuses to call small swings a win will, some months, tell a paying customer that the work has not shown up yet. We would rather say that than sell a rounding error.
Where AI Knows Us fits
AI Knows Us runs 13 agents that do the work end to end: intake, discovery, analyst, diagnosis, planner, writer, editor, publisher, scout, outreach, experiment, lifecycle and the copilot. So it does the fixing as well as the measuring: it writes the pages, publishes them, and asks other sites to name the business. It is made by Clyra Labs in India. Paid plans are not open yet. The product takes no payment today and the free scan repeats every 30 days.
If you want the statistics on their own, read #. If you want the decision framing, read #.
What you can check yourself
You do not need a tool to test this. You need a spreadsheet and an afternoon.
- Write 40 buyer questions that never name your business. For example, "best payroll software for a 20 person agency in India" rather than "is [your brand] good payroll software".
- Pick your engines. Ask every question in a fresh session, with memory and personalisation turned off where the app allows it.
- Record a 1 if your business is named anywhere in the answer, and a 0 if it is not. No partial credit.
- Sum the 1s. Divide by 40. That is your rate, p.
- Run the Wilson formula above, or use a spreadsheet: enter p and n, then compute the centre and half width as two cells.
- Repeat the whole thing after your content work ships, using the same 40 questions and the same engines.
- Compare the two ranges. Overlap means hold your conclusion. No overlap means something real happened.
Doing this once will change how you read any AI visibility dashboard, including ours. If a tool shows you a percentage with no range and no question count, you cannot tell whether its trend line is signal.
Other tools in this category
AI visibility monitoring has several active products. We list them without ranking them against us on measurement method, because we have not audited their statistical reporting and would be guessing.
A fair way to compare any of them, including AI Knows Us, is to ask the vendor three questions: how many questions per run, is the brand name ever included in the prompt, and is a confidence range printed next to the rate.
Who this rule is wrong for
The overlap test is deliberately strict. It will sometimes tell you nothing moved when something small did move. If your work depends on catching small, early shifts quickly, this method will frustrate you, and you should either raise the question count a lot or use a direct test of the difference between two proportions rather than the eyeball overlap check. Teams that need a daily number for a standup will also find monthly ranges too slow. And if your category has so few buyers that 40 realistic questions do not exist, no sampling method will rescue the measurement.
FAQs
How many questions do I need to detect a 10 point improvement?
More than most runs use. At 40 questions, a move from 25 percent to 35 percent leaves ranges of 14 to 40 and 22 to 50, which overlap heavily. Raising the question count narrows both ranges. The practical answer is to either ask several hundred questions or accept that only large moves will be reportable.
Why not just use the simple plus or minus formula from school?
The simple normal approximation misbehaves at small n and at rates near 0 or 100 percent, and can produce bounds below zero or above one. The Wilson score interval stays inside sensible limits. See the NIST e-Handbook section on confidence intervals for a proportion.
Does overlap always mean there is no real difference?
No. Overlap is a conservative signal. Schenker and Gentleman showed that two intervals can overlap while a direct test of the difference still finds it significant. We treat overlap as a reason not to claim a win, not as proof that nothing happened.
Why are questions asked without naming the business?
Naming the business inside the prompt primes the engine to talk about it, so the answer tells you nothing about whether a cold buyer would have been shown that business. A primed reading is not comparable to a blind one, and mixing the two breaks the trend line.
Why ask five AI apps instead of one?
ChatGPT, Gemini, Perplexity, Claude and Google AI Overviews answer the same question from different sources, often on the same day. A single engine reading can move for reasons that have nothing to do with your content.
What does AI Knows Us cost today?
Paid plans are not open yet. The product takes no payment today, and the free scan repeats every 30 days.
Frequently asked questions
How many questions do I need to detect a 10 point improvement?
More than most runs use. At 40 questions, a move from 25 percent to 35 percent leaves ranges of 14 to 40 percent and 22 to 50 percent, which overlap heavily. Raising the question count narrows both ranges. In practice you either ask several hundred questions or accept that only large moves will be reportable.
Why not just use the simple plus or minus formula from school?
The simple normal approximation misbehaves at small sample sizes and at rates near 0 or 100 percent, and can produce bounds below zero or above one. The Wilson score interval stays inside sensible limits. See the NIST/SEMATECH e-Handbook section on confidence intervals for a proportion.
Does overlap always mean there is no real difference?
No. Overlap is a conservative signal. Schenker and Gentleman (The American Statistician, 2001) showed that two intervals can overlap while a direct test of the difference still finds it significant. Treat overlap as a reason not to claim a win, not as proof that nothing happened.
Why are questions asked without naming the business?
Naming the business inside the prompt primes the engine to talk about it, so the answer tells you nothing about whether a cold buyer would have been shown that business. A primed reading is not comparable to a blind one, and mixing the two breaks the trend line.
Why ask five AI apps instead of one?
ChatGPT, Gemini, Perplexity, Claude and Google AI Overviews answer the same question from different sources, often on the same day. A single engine reading can move for reasons that have nothing to do with your content.
What does AI Knows Us cost today?
Paid plans are not open yet. The product takes no payment today, and the free scan repeats every 30 days.