Googlebot: what the crawler is, how it behaves on your site, and how to verify a request really came from Google

Googlebot is the web crawler Google Search uses to discover and fetch pages. Google documents it as one generic name covering two crawler types: Googlebot Smartphone, which simulates a user on a mobile device, and Googlebot Desktop. Both obey the same robots.txt product token, Googlebot, so you cannot allow one and block the other through robots.txt. For most sites Google indexes the mobile version of the content, so most crawl requests arrive from the smartphone crawler and a minority from the desktop one. All of that is from Google's own Googlebot documentation, last updated 3 February 2026.

Googlebot finds new URLs mainly from links on pages it has already crawled, supplemented by sitemaps you submit. That is why an orphan page with no internal link pointing at it can sit unvisited for a long time while the rest of the site is crawled normally.

What Google publishes about Googlebot's behaviour

• Request rate. For most sites Googlebot should not access the site more than once every few seconds on average, and short bursts can look higher than that. If your server is struggling, Google documents a way to reduce the crawl rate.
• Fetch size limits. When crawling for Search, Googlebot crawls the first 2MB of a supported file type and the first 64MB of a PDF. Each referenced resource, such as a CSS or JavaScript file, is fetched separately and is bound by the same limit. Past the cutoff the fetch stops and only the downloaded part is passed on for indexing, and the limit applies to uncompressed data.
• Other crawlers, other limits. Googlebot Image and Googlebot Video are separate crawlers and may have different limits.
• Time zone. When crawling from IP addresses in the United States, Googlebot's time zone is Pacific Time, which matters if you schedule content by date.

The practical reading of the size limits: keep the HTML document itself lean. A page that ships a very large inline payload risks having the end of the document cut off before it is considered, and the fix is ordinary engineering rather than anything clever.

Controlling Googlebot, and the distinction people get wrong

Crawling and indexing are two different things, and the control for each is different. Google states this plainly: blocking Googlebot from crawling a page does not prevent that page's URL from appearing in search results.

What you want	The right control	What it does not do
Stop Googlebot fetching a page	A disallow rule in robots.txt	Does not remove the URL from results, and Google cannot see a noindex tag on a page it is not allowed to fetch
Keep a page out of the index	A noindex robots meta tag or X-Robots-Tag header, on a page that is crawlable	Does not stop the fetch, so the page still consumes crawl requests
Make a page inaccessible entirely	Authentication, such as password protection	Removes it from users as well as crawlers
Reduce load during an incident	Reduce the crawl rate using Google's documented method	Is not a long term indexing strategy

Blocking Googlebot affects Google Search including Discover and Search features, and also Google Images, Google Video and Google News. That breadth is the reason a blanket disallow added during a migration causes problems nobody connects to the migration for weeks.

Verifying Googlebot, which you should do before blocking anything

Google warns that the Googlebot user agent string is frequently spoofed by other crawlers, so a log line claiming to be Googlebot is not proof. The documented way to verify is a reverse DNS lookup on the source IP of the request, or matching the source IP against Google's published Googlebot IP ranges.

Do this before you act on a traffic spike. A scraper wearing the Googlebot name is a security and bandwidth problem, and blocking the real Googlebot because of it is a far more expensive mistake.

How to check what Googlebot is actually doing on your site

• Search Console, Crawl stats. Requests over time, by response code, by file type and by purpose, which is where a rise in failed requests shows up first.
• URL Inspection. Fetches a single URL as Google and shows the crawled page, the rendered HTML and the indexing decision, which settles most arguments about whether a template renders.
• Server logs. The only complete record. Filter by verified Googlebot IPs, then look at which directories and parameter patterns consume the most requests.
• The robots.txt report. Confirms which file Google last fetched and whether it parsed, which catches the deploy that shipped a staging robots.txt to production.

Where requests are going somewhere worthless, the fix is usually internal linking, canonicals and parameter handling rather than a crawl rate change. The waste patterns and the right tool for each are covered in crawl budget optimization, and the stages that follow the fetch are set out in how search engines work.

The other crawlers in your logs

Googlebot is no longer the only one worth reading your access log for. Google operates separate crawlers for Images, Video, News and AdsBot, and the assistant era has added more: crawlers that fetch pages to train models, and fetchers that retrieve a page in real time because a user asked a question. They use different tokens, they are governed by separate robots.txt rules, and treating them as one group is how sites end up blocking the fetcher that would have cited them. Which ones to allow, and a tested way to check what each can actually read, is covered in which AI crawlers should you allow.

What site owners get wrong about Googlebot

Is Googlebot one crawler or two?

One name, two crawler types: Googlebot Smartphone and Googlebot Desktop. You can tell them apart from the user agent header in the request, but not control them separately in robots.txt, because both obey the same token.

Does Googlebot execute JavaScript?

Yes, pages are rendered, and each referenced resource is fetched separately and subject to the same size limit as other files. Compare the raw HTML with the rendered HTML in URL Inspection when content appears on screen but not in the index.

How often does Googlebot visit?

It varies by site and by URL, and Google's guidance is that for most sites it should not be more than once every few seconds on average across the whole site. Pages that change often and are linked prominently get revisited sooner than pages nothing points to.

Can I stop a page appearing in results with robots.txt?

No, and this is the most expensive misunderstanding in technical SEO. A disallowed URL can still appear in results, and because Google cannot fetch the page it also cannot see a noindex tag on it. Allow the crawl and use noindex.

Something in my logs says it is Googlebot. Is it?

Verify it. Run a reverse DNS lookup on the source IP, or check the IP against Google's published Googlebot ranges. The user agent string is spoofed often enough that it should never be treated as identification.

Will blocking Googlebot hide a site from the web?

It will not. Google notes that keeping a site secret by not publishing links to it is close to impossible, because a single outbound click can expose the URL in another site's referrer log. Use authentication for anything that must stay private.

The crawl that decides what an assistant says about you

Everything above is about a crawler whose output you can inspect: you can see the fetch in your logs, the rendered page in Search Console and the result in the index. The newer crawlers produce no such trail. When somebody asks an assistant which supplier to use, the answer is built partly from pages fetched at the moment of the question and partly from third party sources, and nothing in Search Console reports whether your page was one of them. So the access checks worth running now are two: whether Googlebot can fetch and render your pages, and whether the named assistant fetchers can too, followed by whether the answers they produce actually mention you. AI Knows Us runs the second half of that, checking which assistants name a business when buyers ask and listing the pages each answer drew on. You can run it free at aiknowsus.com.