How to tell whether your website is readable by AI crawlers

Why this step and not another

People usually start in the wrong place. They write a long FAQ page, or they buy a tool that counts how often an assistant mentions them, and nothing moves. The reason is dull: the crawler never got the page, or it got a shell of HTML with no words in it.

Crawl access is the cheapest thing to fix and the easiest thing to prove. It is a yes or no answer per crawler, you can test it yourself, and the fix is usually one line in a text file or one toggle in a CDN dashboard. Everything downstream (being mentioned, being cited, being recommended) depends on it. So do this before you commission content, before you buy tracking, before you argue about brand voice.

There is a second reason to start here. In the last two years the default changed. Cloudflare shipped a one click block for AI scrapers on 03 July 2024, and on 01 July 2025 it announced that new domains signing up would be asked whether to block AI crawlers by default. Plenty of marketing teams are blocked today by a setting an engineer never mentioned.

The crawlers worth testing, and who runs them

These are the named user agents the operators themselves document. The point of the table is that you test against real strings from primary sources, not a list someone pasted into a blog post.

One nuance that catches people out. Google documents Google-Extended as a control for Gemini apps and Vertex AI grounding, not as a crawler in its own right, and blocking it does not remove your pages from Google Search. AI Overviews draws on the Search index, so the levers there are Googlebot access and the snippet controls such as nosnippet and max-snippet. Read Google's own page before you change anything.

The rules you write are governed by the Robots Exclusion Protocol, published as RFC 9309 in September 2022. Matching is per user agent group, the most specific matching group wins, and an Allow line can override a broader Disallow. If your robots.txt has grown over five years, assume it contains at least one rule nobody remembers writing.

How to do it: eight steps

1. Fetch your robots.txt as a plain client. Run curl -sS -D - https://example.com/robots.txt -o robots.txt. You want HTTP 200 and a content type of text/plain. A 404 is fine (it means no restrictions). A 403, a redirect chain, an HTML error page or a Cloudflare challenge page is a problem, because a crawler that cannot read the file may treat the whole site as off limits.
2. Read the file for each user agent in the table. Search for User-agent: *, then for each named bot. Remember that a named group replaces the wildcard group for that bot, it does not add to it. Write down one line per crawler: allowed, blocked, or partly blocked.
3. Test a real page with each user agent string. For example: curl -sS -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot" -o /dev/null -w "%{http_code}\n" https://example.com/pricing. Repeat for ClaudeBot, PerplexityBot, OAI-SearchBot and ChatGPT-User. You are looking for 200. A 403 or 429 means something above your application is filtering by user agent, usually a WAF, a bot management product or a rate limit.
4. Open your CDN and WAF settings and look for an AI bot control. On Cloudflare this sits under bot management and related AI crawler controls, and there is also a managed robots.txt feature. On other providers look for "bot fight", "known bots", "scraper protection" or a managed rule set. Record the state of each toggle with today's date, because these defaults change.
5. Check whether your words exist without JavaScript. Run curl -sS https://example.com/pricing | grep -ci "your key phrase". If the count is zero but the phrase is visible in a browser, your content is rendered client side. Vercel's analysis of crawler traffic across its network, published in December 2024, reported that the AI crawlers it observed did not execute JavaScript, so client rendered text was not seen. Treat server rendered HTML as the safe assumption.
6. Check status codes, canonicals and noindex on the pages that matter. curl -sSI https://example.com/pricing should show 200, not 301 into a loop and not 404. Then grep the HTML for noindex, nosnippet and rel="canonical". A page that points its canonical at a different URL is asking to be ignored.
7. Grep your server logs for the real thing. Access is theory until a crawler turns up. Example: grep -iE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|PerplexityBot" access.log | awk '{print $1, $12}' | sort | uniq -c | sort -rn | head -20. Adjust the field numbers to your log format.
8. Verify the hits are genuine. User agent strings are trivial to fake. OpenAI publishes IP ranges at openai.com/gptbot.json, openai.com/searchbot.json and openai.com/chatgpt-user.json. Perplexity documents its verification method on its bots page. Google documents reverse DNS verification for Googlebot on its verification page. Check a sample of ten IPs per crawler, not all of them.

A worked example you can repeat

This is an illustration using the placeholder domain example.com. Swap in your own domain and your own key phrase. Numbers shown are sample output, not a measurement of any real site.

1. robots.txt fetch: curl -sS -o /dev/null -w "%{http_code} %{content_type}\n" https://example.com/robots.txt returns 200 text/plain. Pass.
2. File contents show User-agent: * then Disallow: /wp-admin/, and further down User-agent: GPTBot then Disallow: /. That single block was added during a 2024 scraping scare. GPTBot: blocked. Everything else: allowed.
3. User agent test on the pricing page returns 200 for ClaudeBot and PerplexityBot, and 403 for ChatGPT-User. The 403 is not in robots.txt, so it is coming from the firewall.
4. CDN dashboard shows an AI crawler blocking rule switched on. Date it, screenshot it, decide deliberately whether you want it on.
5. JavaScript test: curl -sS https://example.com/pricing | grep -ci "per seat per month" returns 0, while the phrase is clearly visible in Chrome. The pricing table is client rendered. That is the biggest single finding of the run.
6. Fix list, in order of value: server render the pricing table, remove the GPTBot Disallow: / line, allow the OpenAI and Anthropic live fetch agents through the firewall.
7. Fourteen days later the log grep shows 38 GPTBot hits, 12 OAI-SearchBot hits and 9 PerplexityBot hits, against 0 for all three in the fortnight before. Nine sampled IPs out of ten match the published OpenAI ranges, and the tenth is a scraper pretending to be GPTBot, which you now block.

Total time for a single site with shell access: roughly twenty minutes for the audit, longer for the rendering fix. Keep the output in a dated text file. The value of this test is almost entirely in comparing two dates.

How to tell it worked

Three checks, in order of how quickly they respond.

1. Same day: the access checks pass

• robots.txt returns 200 as text/plain, and no named AI crawler has a Disallow: / you did not intend.
• Every user agent you care about gets 200 on three representative URLs: your home page, a product or pricing page, and one deep content page.
• curl without JavaScript returns the words a buyer would search for. Grep counts above zero for at least three key phrases per page.

2. Within two to four weeks: crawlers actually arrive

Compare the fortnight before your fix with the fortnight after, using the same grep command and the same log format. You want the hit count to go from zero to something, and you want the verified share of those hits to be high. Do not read a small change as a trend. One week of logs is noise.

3. Weeks to months: the assistants can retrieve you

Crawl access is necessary, not sufficient. The honest follow up test is to ask the assistants questions a buyer would ask, without naming your company, and see whether your pages show up as sources. Naming yourself in the prompt tells the model what answer you want, so the reading stops being evidence. There is a longer walk through of that in #, and a stricter dated version of the access test itself in #.

A reasonable expectation: access fixes show in logs in weeks, and show in answers over months, if the content is worth citing. If your pages are thin, opening the gate changes nothing.

Where AI Knows Us fits

You can run everything above yourself. Nothing on this page needs a vendor. What is harder to do by hand is the part after access: asking the same buyer question across several assistants, repeatedly, and knowing whether a change in the answer is real or just noise.

AI Knows Us measures across ChatGPT, Gemini, Perplexity, Claude and Google AI Overviews, because those five answer the same question differently on the same day from different sources. Every tracking question is asked without naming the business, since naming it primes the engine and the reading becomes worthless. Every rate is shown with a 95 percent range, and when two numbers have overlapping ranges they are reported as not having moved. It also does the fixing as well as the measuring: writing pages, publishing them, and asking other sites to name the business. Thirteen agents do the work: intake, discovery, analyst, diagnosis, planner, writer, editor, publisher, scout, outreach, experiment, lifecycle and the copilot. The product is made by Clyra Labs in India.

On pricing: paid plans are not open yet. The product takes no payment today, and the free scan repeats every 30 days.

Who should not choose AI Knows Us

• If your only problem is the one on this page, a blocked crawler or a client rendered page, fix it with curl and your CDN dashboard and move on.
• If you need an enterprise procurement process, a signed contract and a committed paid tier today, the paid plans are not open yet.
• If you want a single headline percentage with no uncertainty attached, you will find the ranges annoying. They are there because a reading off a handful of questions is not a percentage.

Other tools in this space include Profound, Peec AI, Otterly.AI and Scrunch AI. Check their own sites for current features and pricing, since those change.

FAQ