How Search Engines Work: Crawling, Indexing and Ranking Explained
The three stages of a search engine in plain language, what happens at each one, where pages get lost, and how to check your own site at every stage.
Published by AI Knows Us · Updated 12 October 2026
A search engine works in three stages. First it crawls: automated programs follow links and read sitemaps to discover and download pages. Second it indexes: each downloaded page is rendered, read, understood and stored in a vast database, along with everything the engine worked out about it, including its subject, its language and whether it duplicates another page. Third it serves results: when somebody searches, the engine selects from that stored index, scores every candidate against that exact query and the context of the person asking, and returns an ordered list in a fraction of a second. Google documents these three stages itself, and they happen in that order, which means a problem at an early stage makes everything after it impossible.
That sequence is the most useful thing to understand about search, because it tells you what to check first when a page is not appearing: not its ranking, but whether it was ever crawled and stored at all.
Stage one: crawling
A crawler, which Google calls Googlebot, requests pages the way a browser does. It finds pages to request in three ways: links from pages it already knows, URLs listed in a sitemap you submit, and URLs you ask it to look at directly.
Three things decide whether your page gets crawled.
- A route to it. A page nobody links to and no sitemap lists is a page the crawler has no path to. This is the single most common reason new pages sit untouched.
- Permission. A robots.txt file at the root of your domain tells crawlers which paths they may request. A rule that was added for a staging site and never removed can block a whole section quietly for months.
- Capacity. Crawlers allocate a finite amount of requesting to each site, broadly in proportion to how valuable and responsive the site seems. A site that returns thousands of near identical URLs spends that capacity on waste rather than on its real pages.
That third point is the reason faceted navigation on large sites causes so much trouble: filter combinations can generate an effectively unlimited number of URLs, and a crawler will happily spend its budget on them. Our guide to where crawl requests go and the right tool for each kind of waste covers the fixes, which differ depending on whether you want to block, consolidate or simply speed things up.
Stage two: indexing
Being crawled is not being indexed. After a page is downloaded, the engine renders it, including running JavaScript, works out what it is about, and decides whether to store it. Pages are commonly crawled and then not stored, for reasons that are all fixable.
| Reason a page is not indexed | What it means | The fix |
|---|---|---|
| Duplicate, Google chose a different canonical | Another URL is treated as the real version of this content | Consolidate, or set the canonical you want deliberately |
| Excluded by noindex tag | The page itself asks not to be stored | Remove the tag if the request was unintentional |
| Crawled, currently not indexed | Stored judgement deferred, usually about value or similarity | Make the page substantially better, or merge it |
| Blocked by robots.txt | The crawler was never allowed to read it | Correct the rule, and note that robots.txt controls crawling, not indexing |
| Soft 404 | The page looks empty or like an error to the engine | Add real content, or return a proper status code |
| Redirect | This URL points elsewhere, so the destination is indexed instead | Nothing, if intended |
Two mechanics in that table are worth separating clearly. The robots.txt file controls whether a crawler may request a page. A noindex instruction in the page controls whether an engine may store it. They are different stages, and that is why blocking a page in robots.txt is a poor way to keep it out of results: the engine cannot read the page to see your noindex instruction. If you want a page out of the index, allow it to be crawled and let it say noindex.
Canonical tags are the other mechanic worth learning once, because they decide which of several similar URLs is treated as the real one. Our guide to what canonical means, how to set one and the mistakes that cost pages covers the cases that come up most. For sites where the content is assembled in the browser, how Google renders your site explains what the engine sees and in what order.
Stage three: serving and ranking
When somebody searches, the engine looks only at what it has in the index. It then scores candidates against that query. The signals it uses are many, and Google does not publish a weighted list, but the categories are well documented and sensible.
- Meaning. What the query means, including synonyms, likely misspellings and whether it needs fresh information. The engine interprets before it matches.
- Relevance. Whether the page actually addresses that meaning, judged from the whole page rather than from a repeated phrase.
- Quality. Whether the source appears to be a credible one, which is where links and reputation from the rest of the web carry weight.
- Usability. Whether the page works for the person: it renders on their device, loads quickly, and does not jump about while they read. Google's published Core Web Vitals thresholds are the measurable version of this.
- Context. Location, language, and the device being used. This is why two people searching the same words see different results, and why your own position is not a single number.
The last point has a practical consequence. There is no such thing as "our ranking" as one figure. The same page appears at different positions for different people in different places, so the reliable measure is the average position and the click data in Google Search Console rather than what you see in your own browser. Our guides to the three Core Web Vitals metrics and their thresholds and which Search Console reports matter and what each number measures cover both sides of this.
What happens after the ranking: the results page itself
The ordered list is only part of what the engine returns. A modern results page can include a map with local businesses, images, videos, product listings, questions that expand, and an AI generated summary at the top. Each of those is assembled from different sources, which is why appearing in one does not mean appearing in another.
The practical effect is that a large share of searches are now answered on the results page itself, without a click to any website. That is a change in how to measure success rather than a reason to stop: impressions without clicks still mean your business was shown as an answer. Our page on what zero click search means now covers how to read that pattern and what to do about it.
How to check your own site at each stage
- Crawling. Open yourdomain.com/robots.txt and read it. Then in Google Search Console use the URL Inspection tool on an important page and read the crawl result.
- Indexing. In Search Console, open the Pages report and read the reasons pages are excluded. The reasons in the table above are the ones you will see. This report is the most useful page in the whole tool.
- Rendering. In URL Inspection, view the tested page and the rendered HTML. If your main content is missing there, the engine is not seeing what your visitors see.
- Serving. Use the Performance report and read queries, impressions, clicks and average position. That is the ground truth for how you actually appear.
All four checks are free and together take under an hour on a typical site. If a page is missing entirely, our diagnostic for why a website is not showing up on Google works through the causes in the order they occur.
How other engines and assistants differ
Bing runs the same three stages with its own crawler, Bingbot, and its own webmaster tools, and verifying your site there costs nothing. That matters more than it used to, because several AI assistants draw on Bing's index.
AI assistants add a step rather than replacing the model. They have a trained store of knowledge from a point in time, and most of them also retrieve live pages when a question needs current information. The retrieval step uses a search index, so a page that is unreachable or unindexed is invisible to the assistant as well as to the search engine. The fundamentals carry over: reachable, clear, factual pages that an automated reader can parse.
Questions buyers ask about How Search Engines Work
How long does it take for Google to index a new page?
It ranges from hours to weeks, driven by how often your site is crawled and how clearly the page earns a place. You can shorten it: submit the URL in Search Console, make sure your sitemap includes it, and link to it from a page that is already crawled regularly. That last one is the most effective and the most often forgotten.
What is the difference between crawling and indexing?
Crawling is the engine requesting and downloading your page. Indexing is the engine deciding to store it and what it is about. Crawled and not indexed is a normal status and tells you the page was read and not considered worth storing yet, which is a content question rather than a technical one.
Does robots.txt stop a page appearing in search results?
Not reliably, because it operates at the crawling stage. A blocked URL can still appear if other pages link to it, since the engine knows the URL exists without having read it. To keep a page out of results, allow crawling and use a noindex instruction in the page, or protect it behind authentication.
Why do I see a different ranking than my colleague?
Because results are shaped by location, language, device and search history. Your own browser is the least reliable ranking tool you have. Use the average position in Search Console, which aggregates across everyone who actually saw your page.
How does Google pick what goes in an AI summary at the top?
It draws on pages in its index, which is why the same fundamentals apply: the page has to be crawlable, indexed, clear about what it says and credible on the subject. Google's published guidance for these surfaces points back to the same content and technical essentials rather than to a separate technique.
Do search engines read the whole page?
They download and render the whole page, including content built by JavaScript, and they weigh what they find by where it appears and how the page is structured. Content that only appears after a user interaction, such as behind a click, is seen less dependably than content present when the page renders.
This page explains the mechanism of search. For the checklist of fixes that follow from it, see our technical and Search Console guides linked above.
The engine stage nobody audits
There is now a fourth stage worth knowing about, sitting past serving. When somebody asks ChatGPT, Gemini, Perplexity or Google's AI panel a question like "who should I use for X in Y", the system retrieves pages, reads them, and composes one answer naming two or three businesses. Every stage above still applies, since an unindexed page cannot be retrieved, but the output is different: there is no results page, no position, and no way to see which of your pages were used unless you look. Running your buyers' questions through the assistants once is a genuinely informative hour. AI Knows Us does that across the main assistants and shows which sources each answer was built from.