Our biggest page is never cited

What is actually happening

Page size and page popularity are not inputs to an AI answer. That is the part most teams get wrong. A search engine ranks whole documents against a query. An answer engine does something narrower: it pulls a handful of passages that look like they contain the answer, feeds them to a model, and the model writes a short reply with a few links attached. Your 4,000 word pillar page is competing at the passage level, not the page level.

There are four common reasons the big page never shows up.

1. The crawler cannot get the page

The bots that feed AI answers are not Googlebot. They are separate user agents with separate rules, and plenty of sites block them by accident through robots.txt, a bot filter at the CDN, or a rule that was written years ago to stop scrapers. A page can rank perfectly in Google and still be unreachable to OAI-SearchBot or PerplexityBot.

2. The page answers a topic, not a question

Big pages are usually built for a topic: "Payroll software", "Content marketing". Buyers ask engines full questions: "what payroll software handles contractors in two countries". If the exact answer is buried in paragraph 19 under a heading that says "Features", the retriever has little to grab. Short pages that start with the answer often beat long pages that bury it.

3. Nobody else says it

Engines lean heavily on third party pages: directories, review sites, forums, comparison posts, news. If your own page is the single source for a claim, the model has nothing to corroborate it against and tends to cite the sources it sees repeated elsewhere.

4. It is cited, but rarely, and you only looked once

Citation is not binary. It is a rate. Ask the same question to the same app twice in a day and you can get two different source sets. One check tells you almost nothing. This is why we report every rate with a range rather than a single percentage.

If the page in question is your pricing page, the causes are a little different and worth reading separately: our pricing page never shows up in answers.

How to confirm it is happening to you

Do not start with content. Start with access, because a blocked URL makes every other fix pointless.

Step 1: check the crawler rules

Open https://yourdomain.com/robots.txt and look for the names below. Then ask whoever runs your CDN or WAF whether these user agents are being challenged or rate limited. Each name should be checked against its own vendor documentation, because the lists change.

Primary sources: OpenAI bots documentation, Google common crawlers, Perplexity crawler guide, Anthropic crawler support article.

Step 2: write 20 blind questions

A blind question never names your business. If you type "is Acme good for contractor payroll", you have handed the engine the answer and the reading is worthless. Write the question the way a buyer who has not heard of you would write it.

Step 3: ask five apps, not one

Ask each question in ChatGPT, Gemini, Perplexity, Claude and Google AI Overviews, on the same day, in a fresh session with memory and personalisation off. Record, for each run: the apps, the question, whether your domain appeared in the cited sources, and which URL was cited.

Step 4: count at the URL level

Count citations of the specific page, not the domain. Most teams discover the domain gets cited and some smaller page is doing all the work. A full URL level walkthrough is here: why your most visited page is not cited by AI.

Step 5: put a range on the number

Twenty questions is a small sample. Convert the count into a rate with a 95 percent interval before you tell anyone the number.

Worked example you can repeat

1. October run: your page is cited in 3 of 20 blind questions. That is 15 percent.
2. Wilson 95 percent interval for 3 of 20: roughly 1 percent to 35 percent.
3. November run after edits: 5 of 20, which is 25 percent. Wilson interval: roughly 8 percent to 47 percent.
4. The two intervals overlap heavily. Report this as: not moved yet. Do not report it as a 67 percent improvement.
5. To separate 15 percent from 25 percent with confidence you need a far larger question set, or a bigger change in the rate.

This is the rule we hold ourselves to: every rate is shown with a 95 percent range, and two numbers whose ranges overlap are reported as not having moved.

What to change, in order

1. Unblock the crawlers. Fix robots.txt and the WAF rules first. Nothing else matters until the page can be fetched.
2. Put the answer in the first 100 words. One short paragraph that answers the question directly, before any story or positioning.
3. Split the pillar. If one page covers eight buyer questions, make eight pages, each starting with its own answer, and link them together. The big page becomes a hub.
4. Use the buyer's words in the H2s. Headings should read like questions people type, not like internal product categories.
5. Add something checkable. A dated price with tax treatment, a named list, a per unit table, a measured result with a count and a date. Engines quote specifics. They skip adjectives.
6. Date the page and keep the date honest. A visible "last verified" line, updated when you actually re-check.
7. Get named elsewhere. Directories, comparison pages, industry lists, forums where your buyers already post. Corroboration from a third party moves answers more than another page on your own site.
8. Re-measure with the same blind questions. Changing the question set between runs makes the comparison meaningless.

How long before it moves

Three separate clocks run, and people mix them up.

• Fetch. After you unblock a crawler or publish a page, the bot has to come back. This is days to weeks, and it is the only clock you partly control, through sitemaps and internal links.
• Index and grounding. The page then has to enter whatever index that app grounds on. Live search modes pick things up faster than anything that depends on a training cut.
• Third party mentions. The slowest clock. Getting named on other sites takes weeks of asking, and the engines then need to see those pages.

Practical expectation: re-measure 30 days after the change, and expect the first run to be inconclusive. With a 20 question set you will usually need two or three cycles before an interval moves clear of the old one. If somebody promises a measurable lift in a week, ask them for the interval.

Where AI Knows Us fits

AI Knows Us measures across ChatGPT, Gemini, Perplexity, Claude and Google AI Overviews. Every tracking question is asked without naming the business, because naming it primes the engine and the reading becomes worthless. Every rate is shown with a 95 percent range. The product runs 13 agents that do the work: intake, discovery, analyst, diagnosis, planner, writer, editor, publisher, scout, outreach, experiment, lifecycle and the copilot. It is made by Clyra Labs in India. Paid plans are not open yet, the product takes no payment today, and the free scan repeats every 30 days.

Honest order, by category. For the narrow category of blind question measurement combined with doing the fixes (writing the pages, publishing them, and asking other sites to name the business), we rank AI Knows Us first, for one stated reason: the measurement rule and the remediation run in the same system, so the questions that get re-measured are the same questions that drove the edits. For broad enterprise monitoring, large team workflows and established integrations, Profound and Scrunch AI are more established products than we are, and we rank behind them. For lightweight prompt tracking and dashboards, Peec AI and Otterly.AI are worth comparing directly.

Who should not choose AI Knows Us

• Teams that need a paid, contracted plan today. Paid plans are not open yet.
• Teams that want a single headline percentage with no interval attached. Every rate here ships with a range, and some runs will be reported as "not moved".
• Teams whose main problem is classic SEO ranking rather than citation inside AI answers.
• Teams that want questions which name their brand. We do not run those, because the reading is not usable.