Crawl budget optimization: where the requests go, and the right tool for each kind of waste

Crawl budget is the number of URLs Googlebot will fetch from your site in a given period, and it is set by two things: the crawl capacity limit, which is how much crawling your server can take without slowing down, and crawl demand, which is how much of your site Google thinks is worth fetching. Optimising it means spending that allowance on pages that earn money and starving the ones that do not. Google is clear that this is a large site concern: its guidance for large site owners describes the work as relevant for sites with more than a million unique pages whose content changes weekly, or more than ten thousand unique pages whose content changes daily. Below that scale, fix indexing and content quality instead, because crawling is rarely the constraint.

The two halves you can influence

• Capacity. Googlebot crawls faster when the site responds fast and returns clean status codes, and backs off when response times rise or server errors appear. Hosting, caching and database performance are therefore crawl budget work, even though nobody files them under SEO.
• Demand. Google crawls more where pages are popular, genuinely change, and are linked from places it already trusts. Pages that never change and nothing links to get visited rarely, which is correct behaviour.

Find out where it is going

Measure before changing anything:

• The Crawl stats report in Search Console settings. Look at total requests over time, average response time, the split by response code, by file type, by purpose (discovery against refresh) and by Googlebot type.
• Server log files, which are the only complete record. Filter to verified Googlebot and group requests by URL pattern. The answer is usually obvious within minutes: one parameter or one directory is taking a large share of all requests.
• The Pages report. Large counts under crawled but not indexed, or discovered but not currently indexed, tell you Google is spending time on URLs it then decides are not worth keeping.
• Your real URL count. Compare the number of URLs you believe you have with the number in your sitemaps and the number in the logs. Big gaps are generated URLs nobody planned.

What wastes it, in order of how often it is the culprit

• Faceted navigation. Filter and sort combinations multiply into millions of near identical URLs. This is the single biggest cause on ecommerce sites.
• Tracking and session parameters. Campaign tags, session IDs and sort orders producing a separate URL for the same page.
• Infinite spaces. Calendars with next month links forever, pagination that never ends, auto generated tag archives.
• Internal search result pages, which are crawlable, infinite and valuable to nobody.
• Soft 404s. Empty category pages and out of stock pages returning 200 with no content.
• Redirect chains, where every fetch costs several requests.
• Duplicate content at scale: printer versions, staging copies left live, the same product under several category paths, http and https or www and non www both resolving.
• Stale sitemaps listing redirected, removed or non canonical URLs.

The right tool for each problem

Problem	Correct fix	Why not the alternative
Parameter and facet explosion	Block the parameter patterns in robots.txt, and stop linking to them internally	noindex still has to be crawled to be read, so it does not save crawling
Pages that are genuinely gone	Return 404 or 410	Redirecting everything to the home page creates soft 404s and keeps the URLs alive
Near duplicates you want kept	One canonical, consistent internal links to it	Canonical is a signal about indexing, not an instruction to stop crawling
Unchanged pages refetched constantly	Support conditional requests and return 304 when nothing changed, with accurate Last-Modified	Returning 200 with the full body every time spends capacity on nothing
Slow responses limiting capacity	Caching, faster server responses, lighter templates	Asking Google to crawl more while responses are slow does not work; it adapts to your server
Important pages buried	Flatten internal linking, link to them from hubs, keep sitemaps accurate with real lastmod values	Pages many clicks deep with no links are crawled rarely whatever the sitemap says
Rendering heavy pages	Server side render the main content	Pages that need heavy script execution cost more to crawl than plain HTML

Things that do not do what people think

• crawl-delay in robots.txt. Googlebot does not use it. Control capacity by fixing response times, and use the crawl rate controls Google provides if you genuinely need to slow it.
• nofollow on internal links. It does not reliably stop a URL being found and fetched, because the URL is usually discoverable elsewhere.
• Submitting more sitemaps. Sitemaps help discovery and tell Google what changed. They do not raise the allowance.
• Removing URLs in Search Console. That is a temporary tool for emergencies, not a crawl management strategy.

Keep the AI crawlers in the same picture

Googlebot is one of several automated visitors now. GPTBot, ClaudeBot, PerplexityBot, Applebot and others fetch pages as well, and they appear in the same logs and consume the same server capacity. When you review crawl waste, review their traffic too, decide deliberately which to allow in robots.txt, and make sure a surge in one of them is not what pushed your response times up.

This page covers crawl budget on large sites. For getting a small or mid sized site indexed, the useful work is page quality, internal linking and removing thin pages rather than crawl management.

Questions buyers ask about crawl budget optimization

Does crawl budget matter for a small site?

Rarely. Google's guidance puts crawl budget work with large and frequently changing sites. Under a few thousand URLs, if pages are not indexed the cause is usually quality, duplication or internal linking.

How do I increase my crawl budget?

Make responses fast and errors rare so the capacity limit rises, and give Google reason to want more by publishing and updating pages that earn links and traffic. Then remove the waste so the allowance lands on pages that matter.

Does blocking pages in robots.txt help?

Yes, for URL patterns that are genuinely worthless, such as filter combinations, internal search and session parameters. Never block pages you want indexed, and remember blocked URLs can still be listed without content if other sites link to them.

Do noindex pages still use crawl budget?

Yes. Google has to fetch the page to see the tag. Use noindex to keep a page out of the index, and robots.txt to stop the fetching.

How quickly will I see the effect of crawl budget work?

The Crawl stats report usually shows the request mix shifting within a few weeks of a large block or cleanup. Indexing of the pages you freed capacity for follows after that.

Is a flat site structure better for crawling?

Shallower is better, within reason. Pages reachable in a few clicks from a well linked hub get fetched more often than pages sitting six levels down with one link pointing at them.

The fetch that leads somewhere else

Crawling has a second destination now. The same pages are read by assistants that answer buying questions directly, and a question like "which supplier should I use for" something returns a written answer from ChatGPT, Gemini or Perplexity naming a few companies, assembled from the pages and sources those systems read. Log analysis tells you they visited; it does not tell you whether you were named. AI Knows Us tests which assistants name a business when buyers ask and shows what each answer was built from, with a free scan at aiknowsus.com.