Technical GEO audit for a British site: what 220 dated fetches showed about cookie banners, gated content and rendering

Three things get audited together on a British site, and they fail in three completely different ways. We measured all three on 16 October 2026, across 176 requests to 16 public URLs under 11 user agent strings, plus a second pass over the four that varied, 220 requests in all. 153 of the 176 came back HTTP 200 with the declared target string present in the response body, and not one of those 153 had its content replaced by a cookie or consent interstitial. Every failure was an HTTP 403 refusal, 23 of the 176, decided on the user agent string alone. The ordering is the useful part: the thing most teams worry about first did not appear in the measurement at all, and the thing that did appear lives at the edge rather than in the page.

The three layers, and why each one needs its own check

Can an assistant read our site is really three questions, and each fix is bought from a different person.

• Permission. A rule in your robots file names the agent and the path it may have. In our run, nothing failed here.
• Delivery. The edge returns a 200 rather than a 403, a 429, a challenge or a redirect. In our run every failure sat here.
• Content. The bytes that come back contain the fact you want quoted. This belongs to whoever chose the rendering strategy.

The full protocol and the four layer model it sits in are published in our dated test of robots.txt, HTTP access and named crawlers. This page is a narrower run of it, measured.

How the 220 fetches were run

One HTTP client, curl, with 11 user agent strings, so the string is the only variable. No JavaScript was executed, which is the point rather than a shortcoming: it reproduces what a fetcher that does not render receives. The 11 strings were no User-Agent header at all, curl's own default, a desktop Chrome string, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Googlebot, Google-Extended, and node's default. Google-Extended is a control rather than a crawler, because it is a robots file token governing a use of content and makes no requests of its own.

The 16 URLs were fixed first: ten are the domains cited most often in our own 1,260 reading audit of ChatGPT and Gemini, four are specialist tools in this category, one is a control page of ours. One target string per URL was declared before any request, so nothing could be graded afterwards. Four URLs then answered differently to different agents. A burst of 11 rapid requests can trip a rate limiter in a way that looks identical to an agent rule, so those four were re-run the same day in reversed order with a four second gap, and all 44 statuses agreed. That was not enough, and the correction is the most useful thing on this page. A third reading taken hours later from a different network, seven agents with a three second gap, returned HTTP 200 and the full article from two of the four hosts on almost every agent, desktop Chrome included, and returned an explicit HTTP 429 from one of them under GPTBot. Those two were rate limiters, not agent rules, and a same day reversed pass had reproduced the limiter instead of testing for it. Twenty of our twenty three recorded failures carried a challenge body rather than a refusal, which should have told us sooner. So only the two hosts that refused the same agents on both readings are in the table above, and the rule we now work to is this: a 403 under a bot agent is not an access decision until it repeats in a separate session hours later, and a 429 anywhere in the set puts the whole set in doubt.

Cookie banners did not block anything, and the mechanic explains why

A consent banner is almost always a client side overlay. Scripts paint it on top of a document whose content is already there, so a fetcher that never runs those scripts never meets it, and the text underneath travels in the first response. That is what the measurement shows: in all 153 responses that returned a 200, the declared target string was present.

There is a version that does block, and it is a different mechanism: a server side consent gate redirects to a consent URL and serves nothing until a choice is recorded. That shows as a changed final URL, which we recorded on all 220 requests, and none of the 16 did it. So the audit question is not whether you have a banner, but whether it lives inside the document or in front of it, which you settle by fetching your own page with scripts switched off. Whether you need consent at all is a decision for your own legal advisers; this audit speaks only to whether the words travel in the first response.

The refusals, and the four directions they pointed

Twenty three of the 176 requests were refused with an HTTP 403, and they did not cluster the way the usual advice predicts. Every row below was read on 16 October 2026 and can change on any day.

URL	served the page to	refused with 403
help.openai.com/en/	1 of 11, the Chrome string	10 of 11, including GPTBot, OAI-SearchBot and ChatGPT-User
theverge.com	9 of 11	PerplexityBot and ClaudeBot

There is no single rule here to copy, and that is the finding. These configurations were written at different times by different people, and the only way to know which one you have is to knock on your own door and write down what answered.

Rendering: the first response is the audit

For the 153 successful fetches we also counted the visible text in the first response, scripts never run. Across the ten most cited domains in our own audit the spread runs from 159 units to 8,886 on the same day, and the domain at the bottom was cited in 269 of 1,260 readings. First response volume plainly did not decide those citations.

What it does decide is whether a specific fact you want quoted is there to be quoted. The control page in our own set shows the target state: our 597 reading benchmark article returned 2,255 units of visible text to all 11 agent strings, the same article under GPTBot, ClaudeBot and no agent string at all, because the whole piece is in the server response. Aim for that on the handful of pages carrying the facts you want repeated back to a buyer, which is a different job from rebuilding your application.

The audit, in the order worth running it

• Read your robots file as text, not a plugin's summary of it, and check the rules name the agents you mean. If you are generating or editing it, the settings each tool asks for and how to verify the output are set out in our guide to a robots file generator.
• Fetch six of your own URLs under each agent string you care about, recording every hop's status, the final URL, and whether your target string arrived. Run it from an office connection and a cloud server, because bot managers treat those differently.
• Compare the first response against the rendered page and list every fact that appears only in the second, then fix delivery failures before content ones.

Questions about auditing a British site for AI crawlers

Do cookie banners stop AI crawlers reading our content?

On the 16 URLs we fetched on 16 October 2026, no consent interstitial replaced the content in any of the 153 successful responses. An overlay banner is painted by scripts over text already in the document, and a fetcher that does not run scripts never sees it.

Our robots file allows GPTBot. Why would a fetch still fail?

Permission and delivery are enforced by two different pieces of software, and only the second one can refuse you. All 23 failures in our run were HTTP 403 responses chosen at the edge on the agent string, so the bot rules in your CDN console are where to look before you touch the robots file at all.

Is server side rendering necessary for a British site to be cited?

It is necessary for the facts you want quoted to reach a fetcher that does not render, which is narrower than the claim usually made for it. Volume is not the lever: the ten domains that won our audit arrived with wildly different amounts of readable text in response one, and all ten won anyway. Put your prices, coverage, locations and dates in the first response and let the interactive parts arrive afterwards.

Does a 403 to a named crawler mean somebody chose to block it?

Not reliably. Two of our four varying sites refused a desktop browser string and served every assistant crawler we sent, which is unlikely to be anybody's intention. Bot rules accumulate, and the usual cause is a managed rule set switched on years ago by somebody who has since left. Re-run the audit once a quarter and after every edge change.

What an assistant says about your technical setup, and who gets to check it

The answer an assistant gives about a British supplier is built from whatever it could fetch that day, which is why this is worth running rather than reading about. We asked twice and published both answers in full. The run on 15 October 2026 put 102 buying questions per market to both engines and the run on 16 October put 210, from a set designed eleven days apart from the first, and the share of citations landing on somebody's own vendor or agency site came out at 85.6% and then 82.0%. Review directories took 0.8% and then 0.7%. We sell the measurement and the repair work, which is why this page hands over the procedure and the agent strings instead of a score: a buyer about to pay for technical GEO work should be able to run the first half before the invoice arrives. AI Knows Us, the product behind that work, covers five surfaces. Both audits quoted here were run on two of them, ChatGPT and Gemini, and a technical page claiming five from a dataset of two would fail the standard it is asking you to hold a supplier to.