We rechecked primary sources from Google Crawling Infrastructure and Google Search Central on July 29, 2026. The main crawl budget guide was updated July 22, the canonical guide July 10, and the sitemap guide July 8, 2026. We rechecked the unique-page counts for applicability against the official guide on November 3, 2026.
For most small websites, dedicated crawl budget optimization is rarely necessary. Investigate in depth when a large or frequently updated site shows evidence such as important URLs not being fetched, excessive requests to unwanted parameter URLs, or host availability warnings.
First determine whether crawl budget is really the problem
Google's advanced guide is primarily intended for sites with at least one million unique pages updated weekly, sites with at least 10,000 unique pages updated daily, or sites where Discovered - currently not indexed makes up a large share of URLs in Search Console. These numbers are rough estimates, not exact thresholds.
| Meaning | The set of URLs Google can and wants to crawl for a hostname |
|---|---|
| Components | Crawl capacity limit and crawl demand |
| Main audience | Sites with a million or more unique pages updated weekly, 10,000 or more updated daily, or many URLs classified as Discovered - currently not indexed |
| Main sources | Crawl Stats, URL Inspection, Page Indexing, and verified Googlebot server logs |
| Main improvements | Availability, incorrect blocks, duplicate and faceted URLs, soft 404s, redirects, sitemaps, and HTTP caching |
| Success KPIs | Compare fetch delays and unfetched rates for important URLs, unwanted URL share, 5xx and 429 responses, and response times by URL group |
| Not guaranteed | Indexing, rankings, traffic, or revenue. Additional evaluation follows crawling. |
Crawl budget combines crawl capacity and demand
Crawl budget is not determined only by how much traffic the server can handle. Google combines its crawl capacity limit, which avoids overloading the site, with crawl demand, based on updates, popularity, quality, and uniqueness. Google's crawling infrastructure generally treats each hostname separately, so observe www.example.com and shop.example.com separately.
Crawl capacity limit
The level at which Googlebot can crawl without destabilizing the server. It changes with parallel connections, connection duration, latency, TTFB, 5xx, 429, and DNS or network problems.
Parallel connections × duration × host healthCrawl demand
How much Google wants to fetch and refetch URLs. Popularity, overall user value, unique content, significant updates, and site moves can contribute.
Need for updates × value × uniqueness × discovery signalsCrawling, rendering, indexing, and ranking are separate steps
A crawled page is not guaranteed to be indexed. After fetching, Google renders, consolidates duplicates, assesses quality and usefulness, and selects a canonical URL. Ranking is a later step. Therefore, do not treat more crawls as an SEO result in itself.
First identify whether crawl budget optimization applies
Do not rely on size alone; look for evidence that important URLs are not fetched when needed. Rather than assuming same-day crawling is normal, define acceptable business delays for inventory, news, job deadlines, and similar content.
Both size and symptoms are present
- At least one million unique pages with content changing about weekly
- At least 10,000 unique pages with prices, inventory, or news changing daily
- Discovered - currently not indexed accounts for a large share
- Important URL groups are fetched later than business requirements allow
- Server logs show many requests to filter, sort, or calendar URLs
- Crawl Stats shows 5xx, 429, DNS, connection, or hostload problems
Another explanation is more likely
- Few URLs and fetching occurs within a reasonable time after publication
- Logs show a 200 fetch but Page Indexing shows the URL is not indexed
- Duplicates, thin content, incorrect canonicals, or rendering problems
- No internal links or sitemap entry for the URL
- The only concern is that pages are not crawled daily
- The sole KPI is an increase in total crawls
How to check Search Console and server logs
Do not draw a conclusion from one report alone. Crawl Stats shows site-wide trends; URL Inspection shows Google's information and a live test for an individual URL; Page Indexing shows index status; server logs show actual requests.
Crawl Stats
Review total requests, download size, average response time, response codes, file types, purpose, Googlebot types, host status, and URL examples. It is not a complete URL ledger.
Server logs
Aggregate last fetch times, fetch intervals, status codes, and response times for individual URLs or directories. Verify the crawler rather than trusting its User-Agent alone.
URL Inspection
Check Google's last fetched state, crawlability, canonical URL, fetch time, and rendering result. The live test is not the same as the state of Google's index.
Page Indexing
Review downstream outcomes by URL group, including Discovered - currently not indexed, Crawled - currently not indexed, soft 404s, and duplicates.
| Symptom | Evidence | First action |
|---|---|---|
| Host failures | 5xx, 429, DNS, connections, timeouts, Hostload exceeded | Compare Crawl Stats host status, monitoring, URL Inspection, and logs for the same period. |
| Important URLs not fetched | Published, but no Googlebot request in logs | Check internal links, robots.txt, canonical, sitemap, HTTP status, and mobile links. |
| Requests concentrated on unwanted URLs | Filter, sort, session, calendar, duplicates, soft 404 | Identify URL generation rules and internal link sources, then choose canonical, robots.txt, or 404/410 for the right purpose. |
| Fetched but not indexed | Logs show 200, but Page Indexing shows not indexed | Investigate quality, duplicates, Google's selected canonical, rendering, and indexability. |
| Slow recrawling | Last fetch remains old after an important update | Check accurate lastmod, internal links, HTTP caching, crawl demand, and URL value. |
Aggregate logs by URL group
Overall averages can hide important products that are not fetched while unwanted search-result URLs are crawled heavily. At minimum, group canonical and important URLs, filter and sort URLs, internal search, deleted URLs, soft 404s, static assets, and redirects.
Fix inaccessible content first, then organize URL inventory
Before blocking unwanted URLs in bulk, fix availability problems and misconfigurations that prevent fetching important URLs. Test broad robots.txt or URL-generation changes on a representative directory and prepare a rollback.
Fix 5xx, 429, DNS, and timeouts
When Googlebot detects host trouble, the crawl capacity limit falls. Compare incident times, CDN, WAF, rate limits, and origin load.
Unblock important URLs
Check robots.txt, noindex, canonical, authentication, WAF, mobile links, and rendering assets for misconfiguration.
Reduce generation and internal linking of duplicate or infinite URLs
Classify filter, sort, calendar, session, internal search, and combined-parameter URLs; stop unwanted discovery paths.
Use appropriate HTTP responses and canonicals for deletions, moves, and duplicates
Use a direct 301/308 for moves with a replacement, 404/410 for permanent deletion, and consistent canonicals and internal links for duplicates.
Include canonical URLs and accurate lastmod in sitemaps
List URLs you want in search results, and update lastmod accurately for significant changes such as content, structured data, or major links.
Improve page efficiency and HTTP caching
Reduce TTFB and the cost of rendering resources, images, and JavaScript; return 304 Not Modified correctly for unchanged pages.
Do not confuse robots.txt, noindex, canonical, and 404/410
| Method | Primary purpose | Important caveat |
|---|---|---|
| robots.txt | Stop crawl requests | It does not guarantee removal from search. Do not block URLs on which Google must read noindex. Do not use it to protect secrets. |
| noindex | Exclude from the search index | The URL must be crawlable so Googlebot can read the rule; it is not a direct crawl-stop mechanism. |
| rel=canonical | Indicate the preferred duplicate or similar URL | Redirects and canonicals are strong signals; sitemap inclusion is weaker. Point internal links to the canonical too. |
| 404 / 410 | Indicate permanent deletion with no replacement | Do not redirect to an unrelated homepage. Return an appropriate HTTP status instead of a soft 404. |
| 301 / 308 | Indicate a permanent move | Redirect directly to the final URL and avoid long chains and loops. |
| Authentication | Protect private information | robots.txt is a public file, not access control. |
Diagnose and improve crawl budget in seven steps
Compare the same URL groups, periods, and aggregation rules before and after each change. Crawl requests vary by day, season, and site updates, so do not compare only with the day immediately before a change.
Define important URLs and acceptable crawl times
Group products, articles, jobs, and events, then set a business requirement for how many hours or days after publication or significant changes Googlebot should fetch them.
Classify the URL inventory
Classify canonical and indexable URLs, duplicates, filter, sort, pagination, internal search, deletions, soft 404s, redirects, and static assets. Separate hostnames too.
Save baseline data
Record Crawl Stats requests, response codes, file types, purposes, Googlebot types, and host status. Aggregate last fetches, intervals, and response times by URL group from logs.
Fix availability and configuration errors first
Correct 5xx, 429, DNS, timeouts, WAF, incorrect robots.txt, noindex and canonical rules, broken internal links, and blocked rendering assets.
Reduce generation, discovery, and recrawling of unwanted URLs
Consolidate duplicates to canonical URLs and stop generating and internally linking new duplicates. Use robots.txt for URLs that should not be crawled, 404/410 for permanent deletions, and direct redirects for moves with a replacement.
Improve discovery and recrawl signals
Include only canonical URLs in the sitemap and set accurate lastmod values for significant changes. Use crawlable ordinary links to important pages and preserve equivalent discovery paths on mobile.
Remeasure by URL group and roll back if necessary
Compare fetch delays and unfetched rates for important URLs, unwanted URL share, 5xx and 429 responses, response time, and Page Indexing under the same conditions after changes.
Measure timely fetching of important URLs rather than total crawls
A decrease in total crawls is a failure if important URLs are fetched more slowly. Conversely, even if total crawls increase, timely fetching of updated URLs and fewer unwanted requests and failures may indicate improvement.
| Layer | Key metrics | Assessment |
|---|---|---|
| Fetching | Time to first fetch, last fetch date, fetch interval, and unfetched rate | Did median and slow-tail performance improve for important URL groups? |
| Response | 5xx, 429, DNS, timeout, TTFB, and 304 share | Did failures and server load decrease while important URLs remained reliably available? |
| URL inventory | Request shares for canonical, filter, sort, soft 404, and deleted URLs | Did unwanted URLs decline and the share of important URLs rise? |
| Indexing | Page Indexing, Google-selected canonical, and non-indexing reasons | For fetched URLs, did diagnosis move to quality, duplicates, and rendering? |
| Search outcomes | Impressions, clicks, traffic, and business outcomes | Were other factors separated rather than attributing results to crawling changes alone? |
Common crawl budget misconceptions
More crawling always improves rankings
Crawling is a necessary preliminary step but not a ranking signal. If a URL has been fetched but not indexed, investigate quality, duplicates, and canonical selection rather than crawl count.
crawl-delay controls Googlebot
Google's crawlers do not process the nonstandard crawl-delay rule. Address emergency overcrawling with 429 or 503, and persistent URL problems through design.
Repeated sitemap submission speeds up fetching
There is no need to submit the same unchanged sitemap many times a day. A sitemap is a hint and does not guarantee immediate fetching of every listed URL.
All 4xx responses waste crawl budget
Google says 4xx responses other than 429 do not themselves waste crawl budget. Return a correct 404/410 for a deleted URL with no replacement.
Frequently asked questions about crawl budget
Do small sites need crawl budget optimization?
Usually no dedicated measure is needed. If pages are fetched within a reasonable time after publication, routinely review internal links, an up-to-date sitemap, canonicals, robots.txt mistakes, Page Indexing, and content quality.
Will more crawling improve search rankings?
Not necessarily. Crawling is not itself a ranking signal; rendering, assessment, duplicate consolidation, and indexing decisions follow it. Do not promise ranking gains merely by fixing missed important URLs.
Does noindex save crawl budget?
It is not a direct crawl-stop mechanism: Googlebot must fetch the page to read noindex. Use it to exclude pages from search, and manage generation, internal links, canonical signals, and robots.txt for URLs that need not be crawled.
Does blocking a URL in robots.txt remove it from search results?
Not necessarily. robots.txt controls crawling. If the URL is known through other pages, it may appear without its content being fetched. To exclude it, allow crawling so Google can recognize noindex.
Should every 404 be redirected?
No. Use 301 or 308 for a move with a replacement, and 404 or 410 for permanent deletion without one. Redirecting everything to an unrelated homepage can be treated as a soft 404.
Will a faster server always increase crawling?
No. Faster responses may relieve capacity constraints, but low demand can still keep crawling low. Look for evidence of hostload or availability constraints before scaling.
Can Search Console show a complete crawl history for a specific URL?
No. Crawl Stats shows overall trends and example URLs. Use server logs with verified Googlebot requests to establish fetch times and frequency for specific URLs or directories.
Does Google use sitemap priority and changefreq?
Google ignores priority and changefreq. List canonical URLs you want shown in search, and update lastmod accurately for significant content, structured data, or major link changes.
Official Google sources and verification date
The following primary sources were checked for specifications and decision criteria on July 29, 2026. Google's guidance changes; consult the latest version before making implementation changes.
- Google Crawling Infrastructure: Optimize your crawl budget (updated July 22, 2026)
- Google Search Central: Troubleshoot Google Search crawling errors
- Google Crawling Infrastructure: Managing crawling of faceted navigation URLs
- Google Search Central: Build and submit a sitemap (updated July 8, 2026)
- Google Search Central: Specify a canonical URL (updated July 10, 2026)
- Google Search Central: Block indexing with noindex
- Google Crawling Infrastructure: Useful robots.txt rules
- Google Search Central: Redirects and Google Search
- Google Crawling Infrastructure: Myths and facts about crawling
- Search Console Help: Page indexing report
- Search Console Help: URL Inspection tool
Conclusion: use evidence to remove barriers to fetching important URLs
Crawl budget is not a number every site needs to increase. First assess applicability based on size, update frequency, the share of Discovered - currently not indexed URLs, and delays fetching important URLs.
If applicable, separate crawl capacity from demand and compare Crawl Stats, verified Googlebot logs, URL Inspection, and Page Indexing. Fix availability and configuration first; organize duplicate and faceted URLs, soft 404s, and redirect chains; and include canonical URLs and accurate lastmod in the sitemap.
Whether important URLs were fetched when needed, rather than total crawls, is the most practical success criterion for managing crawl budget.