How AI search engines handle gated content: a bot access audit where only two of four apparent blocks survived a second reading

A paywall is rarely what stops an assistant reading your page. An edge rule is, and establishing that an edge rule exists takes more than one reading. On 16 October 2026 we knocked on 16 public doors with 11 agent names, 176 requests, and 23 were turned away with an HTTP 403. Twelve of the 16 URLs answered all 11 names identically. Four did not, and those four are the ones to report carefully: read again from a second network, two behaved as before and two served content they had previously refused. So the defensible count is not 23 refusals. It is 12 refusals on two hosts that answered the same way every time anybody asked, and 11 statuses on two hosts we will not characterise.

Four things get merged into the word gated

• A hard gate. The server never sends the protected text to anybody not signed in. Nothing in the response to read, by design.
• A metered gate. The server sends the full text and a browser counter decides when to cover it, so a fetcher running no scripts reads all of it.
• An interstitial. A consent or registration overlay painted over content that is already there. Invisible to a fetcher, which never runs the script that paints it.
• An edge decision. A bot rule, a reputation score, a rate limit or a browser challenge, returning a 403 before your application is involved. Every failure in our run came from here, and telling these four apart is the hard part.

Those four need four different fixes from four different people, and one plain HTTP request to a paid page tells you which you have: a 200 carrying the protected text is a metered gate, a 200 without it is a hard gate working as designed, a redirect to a sign in URL is a server side gate, and a 403 is an edge decision that needs the second reading described below. The robots configurations to publish and the fields to log are in our dated robots.txt and access control test. On a hosted platform the gating is the platform's rather than yours, and the controls you get in one common case are in our guide to the built in Squarespace controls.

The measurement, and the three readings it took

The setup: 16 URLs fixed in advance, one declared target string each, requested through one HTTP client under 11 agent strings including GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot and ClaudeBot. The full protocol sits in our technical GEO audit of the same run.

Four URLs answered differently by agent. Establishing what that meant took three more readings, set out in the next section. Two of the four survived, and these are those two.

host, same answer on every reading	served the page to	refused with 403
help.openai.com/en/	1 of 11, the Chrome string	10 of 11, including GPTBot, OAI-SearchBot and ChatGPT-User
theverge.com	9 of 11	PerplexityBot and ClaudeBot

Both rows held on every reading, from both networks. The OpenAI help centre served a desktop Chrome string and nothing else, so the three crawlers OpenAI operates were refused by OpenAI's own help pages, and the body returned asked for JavaScript and cookies, which a fetcher cannot supply. The Verge refused two named assistant crawlers with a plain forbidden page and served five other agents, all three of OpenAI's among them.

The two hosts we removed, and the rule they produced

On the first reading, searchengineinfo.org served the page only to the request sending no agent name and refused the other ten including desktop Chrome, while otterly.ai refused desktop Chrome and served every bot name. Both looked like findings. Both are out of the table, because a second operator reading them from another network got HTTP 200 and the full article on almost every agent, Chrome included, plus an explicit HTTP 429 rate limit where we had recorded a 403.

Our own saved responses should have stopped us sooner. Twenty of the twenty three carried a challenge body rather than a refusal: "Checking your browser before accessing" from searchengineinfo.org, "Enable JavaScript and cookies to continue" from the OpenAI help centre. A status code never says which you have, and we read codes and not bodies. The Verge returned a plain forbidden page with no challenge, and that row has never moved.

So this is the rule, and it costs one afternoon more than most published crawler studies spend.

• A 403 under a bot agent is not an access decision until it reproduces in a separate session hours or days later and from a second network. A repeat pass inside one session reproduces whatever state the first created: ours matched 44 of 44, then 28 of 28 days later, and was still wrong on half the hosts.
• One HTTP 429 anywhere in the set makes the whole set suspect, because a limiter returning 429 to one agent may be returning 403 to the rest.
• Read the body of every non-200 before writing anything down. A challenge body means the status is a challenge, not a policy, and a fetcher failing a JavaScript challenge is a rendering problem wearing an access problem's status code.
• Report the readings that disagreed. Two of our four candidate rows did not survive, and a crawler study that never lost a row has probably read only once.

Those two hosts may be applying an agent rule, a reputation score, a limiter, or all three. Two conflicting readings cannot separate them, so we will not pretend otherwise.

What isAccessibleForFree declares, and what it does not do

Google documents a structured data pattern for paid content. From its documentation on subscription and paywalled content, read 16 October 2026: the markup "helps Google differentiate paywalled content from the practice of cloaking, which violates spam policies", and the guide "only applies to content that you want crawled and indexed".

<script type="application/ld+json">
{ "@context": "https://schema.org", "@type": "Article", "headline": "Your article headline", "datePublished": "2026-10-16", "isAccessibleForFree": false, "hasPart": { "@type": "WebPageElement", "isAccessibleForFree": false, "cssSelector": ".paywall" }
}
</script>

Three rules from it are easy to get wrong. Only class selectors are accepted for cssSelector, content sections must not be nested, and if you do not want the text reaching the browser at all the documentation says to choose an implementation that does not supply the paywalled content to the browser, which is the hard gate rather than the metered one.

The scope is often overstated. This is Google Search documentation about how Google treats a paid page; OpenAI's crawler overview page, read the same day, names four user agents and says nothing about paywall markup. The flag keeps a metered page out of the cloaking category rather than telling five assistants how to treat your article. Which leaves the editorial decision doing the real work: publish the method, the sample size, the date and the headline figure outside the gate as their own readable page, and keep the analysis and the data behind it. An assistant that can quote your figure sends the reader to the gate; one that can read nothing names somebody else.

Questions about paywalls, gated content and AI crawler access

Can AI crawlers read content behind a metered paywall?

If the meter runs in the browser, yes: the full text is in the server response and a fetcher executing no scripts reads all of it. Fetch your own gated URL with a plain HTTP client and search the raw response for a sentence from deep inside the article.

Is it worth putting a disallow rule on our subscriber archive?

Split the question first. A rule aimed at the crawler gathering text in bulk is a decision about how your writing may be used. A rule catching the single fetch behind one person's question is a decision to be absent from that answer. Write down which you meant, then verify at the edge from two networks, because the OpenAI help centre refuses OpenAI's own three crawlers and that is unlikely to be anybody's intention.

Why did a crawler get a 403 when our robots file allows it?

Allowing something and sending it are handled by two different systems, and the one that sends can say no. All 23 of our HTTP 403 responses came from the edge rather than the application, so your CDN bot rules are where to look. Before changing one, read the response body and take a second reading from another network: of our four candidate hosts, only two answered the same way every time.

How many readings before a block is real?

At least two, in separate sessions, from different networks, with the body of every non-200 read rather than only its status code. Our first reading produced four candidate blocks and two of them did not survive that test.

What an assistant says about paid research, and the number we publish about ourselves

Gating matters because an assistant answering a buying question names whichever source it could read, and for paid research that is usually the free summary somebody else published. Our own measurement of who gets named is public, including the part that does not flatter us: across 597 readings of ChatGPT and Gemini on 15 October 2026, 85.6% of citations went to a vendor or agency own site, review directories took 0.8%, and our own domain was cited 3 times. A separately designed run the next day gave 82.0%, 0.7% and 6 citations of us in 1,260 readings. We make money running that measurement for other companies and fixing what it finds, which is why the agent strings, the dates and the two rows we withdrew are here instead of a testimonial. Both runs questioned ChatGPT and Gemini, two of the five surfaces AI Knows Us watches.