Crawlability, the eight things that stop a crawler reaching a page, and how to test each one in under a minute

Crawlability is whether an automated client can request a URL on your site, get a successful response, and read the content in it. That is the whole definition, and the reason it matters is sequencing: a page that cannot be crawled cannot be indexed, and a page that is not indexed cannot rank or be quoted, so every other improvement you make to it is spend with no mechanism. Crawlability is not the same as indexability. A page can be perfectly crawlable and still excluded from the index by a rule you set. The eight blockers below are the ones that actually occur, each with a test you can run in under a minute.

One thing to establish first, because it changes how you read every test. Google documents that its crawlers identify themselves through three things: the HTTP user agent header, the source IP address, and the reverse DNS hostname of that IP, and that you can use those three details to verify a request is genuinely from Google. Read on 31 October 2026. That matters because the first instinct when a page is not being crawled is to check the logs, and a log entry claiming to be a crawler proves nothing on its own.

The eight blockers, in the order they occur in practice

• A robots.txt rule. A disallow that matches the path, often inherited from a staging configuration or written for one folder and matching more than intended. Test: fetch your own /robots.txt and read it in full, then check the specific path against every rule rather than against the one you remember.
• A response code that is not 200. A 404, a 403, a 500, or a redirect chain that ends somewhere unexpected. Test: request the URL and read the status code, not the page. A styled error page returns a body and a status that disagree, and the status is the fact.
• No internal link anywhere on the site. A page reachable only by typing the URL is a page a crawler has no route to. Test: search your own site's rendered HTML for the slug. If nothing links to it, nothing will find it except a sitemap entry.
• A sitemap that does not contain it, on a large site. Not a blocker on its own, and decisive on a site with tens of thousands of URLs where discovery depth matters. Test: search your sitemap for the URL.
• Content that only exists after a script runs. The first response contains a shell and the words arrive later. Test: request the page with a plain client that executes no JavaScript and read what came back in the first response. If the text is missing, that is what some clients will see.
• A login, a paywall or a cookie wall in front of it. Test: open the URL in a private window with no cookies.
• A server or edge rule that refuses non browser clients. Bot protection, rate limiting, or a firewall rule applied more broadly than intended. Test: request the page with a non browser user agent string and compare the status and the body length against a browser request. A 429 anywhere in the results means you are looking at a rate limiter rather than a policy.
• Crawl budget exhausted on pages that do not matter, on large sites only. Google documents that it aims to crawl as many pages as it can on each visit without overloading the server, and that you can reduce the crawl rate if the server struggles. Test: look at where the requests are actually going, which is the subject of where the requests go and the right tool for each kind of waste.

Crawlability, indexability and rankability are three questions

Question	What it asks	Where the answer lives
Crawlable	Can a client request this URL and read the content	robots.txt, the status code, the first response body
Indexable	Is the page allowed into the index, and is it the version chosen	The robots meta tag, the X-Robots-Tag header, the canonical
Rankable	Does it deserve a position for a query	The content, the demand, the competition

Mixing these up wastes more time than any technical fault. The specific confusion to avoid is blocking a page in robots.txt in order to keep it out of the index. Blocking the crawl stops the page being read, which means any noindex rule inside it is never seen, so the URL can persist in results with no description. If you want a page out of the index, let it be crawled and use a noindex rule. If you want it never requested, block it and accept that you have given up control of how it appears.

A ten minute crawlability check for one important page

1. Request it with a plain client and record the status code. Anything other than 200 ends the investigation there.
2. Read the first response body for your own words. Search the raw response for a sentence you know is in the article. If it is absent, the content is arriving after a script and that is the finding.
3. Check robots.txt against this exact path. Not the folder above it.
4. Check the robots meta tag and the X-Robots-Tag header. The header version is invisible in the page source and is the most commonly missed rule on the whole list.
5. Check the canonical. A canonical pointing at another URL is a statement that this page is not the one you want indexed, which is sometimes exactly right and sometimes a template default nobody noticed.
6. Find one internal link pointing at it from a page that is itself crawlable.
7. Repeat steps one and two with a non browser user agent and compare. A difference here is an edge or firewall rule, not a content problem.

Two things worth knowing about crawler identity

Google's own crawler documentation, read on 31 October 2026, notes that AdsBot is a special case which can bypass the global rules written for the wildcard user agent in robots.txt where the ad publisher has agreed to it. So a wildcard disallow is not a universal statement, and a site owner who assumes it is will be surprised by the traffic in the logs. The same documentation names Storebot-Google as caching content only under certain conditions, and Google Site Verifier as a fetcher triggered by a person rather than by a schedule.

The practical consequence is that robots.txt is a set of per agent instructions rather than a single gate, and the only way to know what your file actually permits is to read every rule against every agent you care about. Our own file, with every line published and nothing blocked, is in our robots file and the fourteen agents it does not block, which is a useful reference point because it shows the whole file rather than the part that supports a point. The settings at the edge that help or hinder crawling, on one widely used platform, are set out in the settings that help crawling and the ones that block it.

What to fix first when several things are wrong

Fix in request order, which is the order a crawler meets the obstacles. Robots.txt first, because nothing else is reached while it blocks. Then the status code. Then the first response body. Then the robots meta and the header. Then the canonical. Then internal linking. Then the sitemap. Fixing a canonical on a page that returns a 403 is a change nobody will ever see, and that sequence error accounts for a surprising share of technical work that produces no result.

Questions about crawlability

What does crawlability mean?

Whether an automated client can request a URL, receive a successful response, and read the content it contains. It is the first of three separate questions, the others being whether the page is allowed into the index and whether it deserves a position.

How do I check if a page is crawlable?

Request it with a plain client and read the status code, then search the first response body for a sentence you know is on the page. Those two readings answer most cases in under a minute. After that, check robots.txt against the exact path and look for the X-Robots-Tag header.

What is the difference between crawlability and indexability?

Crawlability is whether the page can be reached and read. Indexability is whether it is permitted into the index once read, which is controlled by the robots meta tag, the X-Robots-Tag header and the canonical. A page can be crawlable and deliberately not indexable.

Does blocking a page in robots.txt remove it from search results?

It stops the page being requested, which means any noindex rule inside it is never read, so the URL can remain in results without a description. To remove a page from the index, allow the crawl and use a noindex rule instead.

Does JavaScript hurt crawlability?

The page is still crawlable; what changes is what arrives in the first response. The reliable approach is to make sure the words you want read are present in the first response rather than assembled afterwards, and the way to confirm it is to read that first response yourself.

How do I confirm a crawler request in my logs is genuine?

Google documents three signals used together: the user agent header, the source IP address and the reverse DNS hostname of that IP. A user agent string on its own is self reported and can say anything.

The same test, run for a reader that does not crawl on a schedule

Everything above is about a client that comes to your site on its own timetable and stores what it finds. There is a second kind of request now, and it behaves differently in one way that matters: it often arrives because somebody has just asked a question, fetches a small number of pages, and reads only what the first response contains. There is no second visit after a script finishes. That makes the fifth blocker on the list above the most consequential of the eight, because a page whose content arrives late is a page that is technically crawlable and practically unreadable to that client. The check is the same check, run with the agents those systems use, and the useful output is a list of pages whose first response is empty. AI Knows Us reads pages the way those clients do and reports which of the assistants' answers your pages are actually being used in.

How we know this

We read the assistants rather than guessing what they do. In our own audit of 1,260 saved answers from ChatGPT and Gemini across the United States, the United Kingdom and India, 82.0% of every citation landed on a site the business itself owned, and review directories took 0.7%. Counted as one domain per reading, 16,516 of 20,143.

That is the whole asymmetry: the pages you own are the pages an assistant quotes back to a buyer. See the dated readings and the method on our proof page.

Find out whether assistants name your business, on your own domain and your own buying questions.

By Aditya Goel, Founder, AI Knows Us. Reviewed 1 November 2026.