ServerField Server operations and website growth
Technical SEO Googlebot and log analysis Updated July 29, 2026

What is crawl budget?
How to diagnose and improve it

The goal is not to increase crawl counts for their own sake. This guide walks through determining whether the issue applies, crawl capacity and demand, URL inventory, Search Console and server logs, and improvement priorities so important URLs are fetched when needed.

Published: January 8, 2025 Last updated: July 29, 2026 Written and reviewed by: Finite Field Editorial Team
Crawl diagnostic overview
Important URLs fetched96.4%Within 7 days
Unnecessary URL share18.7%Needs classification
Host availabilityHealthyLow 5xx

These are illustrative diagnostic items, not measured results. Use Crawl Stats, URL Inspection, Page Indexing, and verified Googlebot logs for their respective purposes.

We rechecked primary sources from Google Crawling Infrastructure and Google Search Central on July 29, 2026. The main crawl budget guide was updated July 22, the canonical guide July 10, and the sitemap guide July 8, 2026. We rechecked the unique-page counts for applicability against the official guide on November 3, 2026.

For most small websites, dedicated crawl budget optimization is rarely necessary. Investigate in depth when a large or frequently updated site shows evidence such as important URLs not being fetched, excessive requests to unwanted parameter URLs, or host availability warnings.

The answer in 30 seconds

First determine whether crawl budget is really the problem

Google's advanced guide is primarily intended for sites with at least one million unique pages updated weekly, sites with at least 10,000 unique pages updated daily, or sites where Discovered - currently not indexed makes up a large share of URLs in Search Console. These numbers are rough estimates, not exact thresholds.

More likely to applyLarge e-commerce, jobs, real estate, news, or user-generated content sites, and sites with frequently changing inventory
Evidence to check firstDelays fetching important URLs; 5xx, 429, or Hostload exceeded; many requests to unwanted URLs
Success measureTimely fetching of important URLs and a lower share of unwanted URLs, rather than total crawls
MeaningThe set of URLs Google can and wants to crawl for a hostname
ComponentsCrawl capacity limit and crawl demand
Main audienceSites with a million or more unique pages updated weekly, 10,000 or more updated daily, or many URLs classified as Discovered - currently not indexed
Main sourcesCrawl Stats, URL Inspection, Page Indexing, and verified Googlebot server logs
Main improvementsAvailability, incorrect blocks, duplicate and faceted URLs, soft 404s, redirects, sitemaps, and HTTP caching
Success KPIsCompare fetch delays and unfetched rates for important URLs, unwanted URL share, 5xx and 429 responses, and response times by URL group
Not guaranteedIndexing, rankings, traffic, or revenue. Additional evaluation follows crawling.

Crawl budget combines crawl capacity and demand

Crawl budget is not determined only by how much traffic the server can handle. Google combines its crawl capacity limit, which avoids overloading the site, with crawl demand, based on updates, popularity, quality, and uniqueness. Google's crawling infrastructure generally treats each hostname separately, so observe www.example.com and shop.example.com separately.

Crawl capacity limit

The level at which Googlebot can crawl without destabilizing the server. It changes with parallel connections, connection duration, latency, TTFB, 5xx, 429, and DNS or network problems.

Parallel connections × duration × host health

Crawl demand

How much Google wants to fetch and refetch URLs. Popularity, overall user value, unique content, significant updates, and site moves can contribute.

Need for updates × value × uniqueness × discovery signals
Important: A fast server does not increase crawling when demand is low. Conversely, infrastructure improvements may help if important high-quality content is not fetched and evidence shows the server is at its capacity limit.

Crawling, rendering, indexing, and ranking are separate steps

A crawled page is not guaranteed to be indexed. After fetching, Google renders, consolidates duplicates, assesses quality and usefulness, and selects a canonical URL. Ranking is a later step. Therefore, do not treat more crawls as an SEO result in itself.

First identify whether crawl budget optimization applies

Do not rely on size alone; look for evidence that important URLs are not fetched when needed. Rather than assuming same-day crawling is normal, define acceptable business delays for inventory, news, job deadlines, and similar content.

Consider a full diagnosis

Both size and symptoms are present

  • At least one million unique pages with content changing about weekly
  • At least 10,000 unique pages with prices, inventory, or news changing daily
  • Discovered - currently not indexed accounts for a large share
  • Important URL groups are fetched later than business requirements allow
  • Server logs show many requests to filter, sort, or calendar URLs
  • Crawl Stats shows 5xx, 429, DNS, connection, or hostload problems
Prioritize ordinary SEO

Another explanation is more likely

  • Few URLs and fetching occurs within a reasonable time after publication
  • Logs show a 200 fetch but Page Indexing shows the URL is not indexed
  • Duplicates, thin content, incorrect canonicals, or rendering problems
  • No internal links or sitemap entry for the URL
  • The only concern is that pages are not crawled daily
  • The sole KPI is an increase in total crawls
1. Define important URLsIdentify URL groups such as products, articles, jobs, and events, and acceptable fetch delays.
2. Verify missing fetchesCheck Googlebot fetch times in server logs; do not confuse crawling with Page Indexing.
3. Isolate the causeSeparate capacity, demand, discovery, incorrect blocks, quality, and duplicate-content issues.
For small sites: It is not unusual for discovery to take several days or more after publication. Unless a site handles time-sensitive information, do not assume a missing same-day fetch is abnormal.

How to check Search Console and server logs

Do not draw a conclusion from one report alone. Crawl Stats shows site-wide trends; URL Inspection shows Google's information and a live test for an individual URL; Page Indexing shows index status; server logs show actual requests.

CS

Crawl Stats

Review total requests, download size, average response time, response codes, file types, purpose, Googlebot types, host status, and URL examples. It is not a complete URL ledger.

LOG

Server logs

Aggregate last fetch times, fetch intervals, status codes, and response times for individual URLs or directories. Verify the crawler rather than trusting its User-Agent alone.

URL

URL Inspection

Check Google's last fetched state, crawlability, canonical URL, fetch time, and rendering result. The live test is not the same as the state of Google's index.

PI

Page Indexing

Review downstream outcomes by URL group, including Discovered - currently not indexed, Crawled - currently not indexed, soft 404s, and duplicates.

Common symptoms, evidence, and first actions
SymptomEvidenceFirst action
Host failures5xx, 429, DNS, connections, timeouts, Hostload exceededCompare Crawl Stats host status, monitoring, URL Inspection, and logs for the same period.
Important URLs not fetchedPublished, but no Googlebot request in logsCheck internal links, robots.txt, canonical, sitemap, HTTP status, and mobile links.
Requests concentrated on unwanted URLsFilter, sort, session, calendar, duplicates, soft 404Identify URL generation rules and internal link sources, then choose canonical, robots.txt, or 404/410 for the right purpose.
Fetched but not indexedLogs show 200, but Page Indexing shows not indexedInvestigate quality, duplicates, Google's selected canonical, rendering, and indexability.
Slow recrawlingLast fetch remains old after an important updateCheck accurate lastmod, internal links, HTTP caching, crawl demand, and URL value.

Aggregate logs by URL group

Overall averages can hide important products that are not fetched while unwanted search-result URLs are crawled heavily. At minimum, group canonical and important URLs, filter and sort URLs, internal search, deleted URLs, soft 404s, static assets, and redirects.

Log analysis: User-Agents claiming to be Googlebot can be spoofed. Include official IP range or DNS verification in operations. Align aggregation units, time zones, and differences in CDN logging with Crawl Stats.

Fix inaccessible content first, then organize URL inventory

Before blocking unwanted URLs in bulk, fix availability problems and misconfigurations that prevent fetching important URLs. Test broad robots.txt or URL-generation changes on a representative directory and prepare a rollback.

  1. Fix 5xx, 429, DNS, and timeouts

    When Googlebot detects host trouble, the crawl capacity limit falls. Compare incident times, CDN, WAF, rate limits, and origin load.

  2. Unblock important URLs

    Check robots.txt, noindex, canonical, authentication, WAF, mobile links, and rendering assets for misconfiguration.

  3. Reduce generation and internal linking of duplicate or infinite URLs

    Classify filter, sort, calendar, session, internal search, and combined-parameter URLs; stop unwanted discovery paths.

  4. Use appropriate HTTP responses and canonicals for deletions, moves, and duplicates

    Use a direct 301/308 for moves with a replacement, 404/410 for permanent deletion, and consistent canonicals and internal links for duplicates.

  5. Include canonical URLs and accurate lastmod in sitemaps

    List URLs you want in search results, and update lastmod accurately for significant changes such as content, structured data, or major links.

  6. Improve page efficiency and HTTP caching

    Reduce TTFB and the cost of rendering resources, images, and JavaScript; return 304 Not Modified correctly for unchanged pages.

Do not confuse robots.txt, noindex, canonical, and 404/410

Choose URL controls by purpose
MethodPrimary purposeImportant caveat
robots.txtStop crawl requestsIt does not guarantee removal from search. Do not block URLs on which Google must read noindex. Do not use it to protect secrets.
noindexExclude from the search indexThe URL must be crawlable so Googlebot can read the rule; it is not a direct crawl-stop mechanism.
rel=canonicalIndicate the preferred duplicate or similar URLRedirects and canonicals are strong signals; sitemap inclusion is weaker. Point internal links to the canonical too.
404 / 410Indicate permanent deletion with no replacementDo not redirect to an unrelated homepage. Return an appropriate HTTP status instead of a soft 404.
301 / 308Indicate a permanent moveRedirect directly to the final URL and avoid long chains and loops.
AuthenticationProtect private informationrobots.txt is a public file, not access control.

First decide whether filtered and sorted URLs should appear in search

Faceted navigation on e-commerce, jobs, real estate, and event sites can create enormous URL spaces through combinations of color, size, price, location, and sort order. A crawler cannot fully judge value before fetching, potentially causing excessive crawling of unwanted URLs and delays discovering new ones.

Do not target search

Design to reduce crawling itself

  • Avoid generating large numbers of unwanted parameter URLs as internal links
  • Block clear patterns with robots.txt
  • Represent filter state with URL fragments when appropriate
  • Keep ordinary links to representative listings and detail pages
Target search

Keep the URL space finite and consistent

  • Standardize parameter order and spelling
  • Consolidate combinations returning the same results
  • Return appropriate status for empty results and invalid values
  • Use internal links and sitemaps to expose only valuable combinations
User-agent: Googlebot
Disallow: /*?*color=
Disallow: /*?*size=
Disallow: /*?*sort=

# Test against the actual URL rules to avoid blocking important pages
Do not rely on canonical alone: Google says canonical or nofollow may help lower faceted URL crawling over time, but robots.txt or URL design is generally more effective for URLs that should not be crawled. Do not block combinations that have genuine search demand indiscriminately.

Diagnose and improve crawl budget in seven steps

Compare the same URL groups, periods, and aggregation rules before and after each change. Crawl requests vary by day, season, and site updates, so do not compare only with the day immediately before a change.

Define important URLs and acceptable crawl times

Group products, articles, jobs, and events, then set a business requirement for how many hours or days after publication or significant changes Googlebot should fetch them.

Priority URL groupsAcceptable delayPublished or updated atResponsible team

Classify the URL inventory

Classify canonical and indexable URLs, duplicates, filter, sort, pagination, internal search, deletions, soft 404s, redirects, and static assets. Separate hostnames too.

URL generation rulesInternal link sourcesHTTP statusCanonical destination

Save baseline data

Record Crawl Stats requests, response codes, file types, purposes, Googlebot types, and host status. Aggregate last fetches, intervals, and response times by URL group from logs.

Same time zoneSame periodMedian and 95th percentileCounts by URL group

Fix availability and configuration errors first

Correct 5xx, 429, DNS, timeouts, WAF, incorrect robots.txt, noindex and canonical rules, broken internal links, and blocked rendering assets.

URL InspectionActual HTTP responseMonitoring logsCompare CDN and origin

Reduce generation, discovery, and recrawling of unwanted URLs

Consolidate duplicates to canonical URLs and stop generating and internally linking new duplicates. Use robots.txt for URLs that should not be crawled, 404/410 for permanent deletions, and direct redirects for moves with a replacement.

Faceted URLsSoft 404Redirect chainsSession URLs

Improve discovery and recrawl signals

Include only canonical URLs in the sitemap and set accurate lastmod values for significant changes. Use crawlable ordinary links to important pages and preserve equivalent discovery paths on mobile.

Canonical URLsAccurate lastmodInternal linksMobile links

Remeasure by URL group and roll back if necessary

Compare fetch delays and unfetched rates for important URLs, unwanted URL share, 5xx and 429 responses, response time, and Page Indexing under the same conditions after changes.

Success criteriaStop conditionsRollback procedureNext hypothesis

Measure timely fetching of important URLs rather than total crawls

A decrease in total crawls is a failure if important URLs are fetched more slowly. Conversely, even if total crawls increase, timely fetching of updated URLs and fewer unwanted requests and failures may indicate improvement.

Median and P95Time from publication or update to first Googlebot fetch
Important URL fetch rateShare fetched within the acceptable time
Unwanted URL shareShare of Googlebot requests for filter, duplicate, and soft 404 URLs
5xx and 429Counts and times of host failures and rate limiting
Response timeReview the distribution and slow periods, not only the average
Downstream outcomesTrack Page Indexing, search impressions, and traffic separately
Metrics to compare before and after changes
LayerKey metricsAssessment
FetchingTime to first fetch, last fetch date, fetch interval, and unfetched rateDid median and slow-tail performance improve for important URL groups?
Response5xx, 429, DNS, timeout, TTFB, and 304 shareDid failures and server load decrease while important URLs remained reliably available?
URL inventoryRequest shares for canonical, filter, sort, soft 404, and deleted URLsDid unwanted URLs decline and the share of important URLs rise?
IndexingPage Indexing, Google-selected canonical, and non-indexing reasonsFor fetched URLs, did diagnosis move to quality, duplicates, and rendering?
Search outcomesImpressions, clicks, traffic, and business outcomesWere other factors separated rather than attributing results to crawling changes alone?
Rollback conditions: Revert if important URLs are blocked by robots.txt, rendering resources become inaccessible, fetch delays worsen, incorrect 404s, 410s, or canonicals increase, or indexable URLs disappear from the sitemap.

Common crawl budget misconceptions

Misconception

More crawling always improves rankings

Crawling is a necessary preliminary step but not a ranking signal. If a URL has been fetched but not indexed, investigate quality, duplicates, and canonical selection rather than crawl count.

Misconception

crawl-delay controls Googlebot

Google's crawlers do not process the nonstandard crawl-delay rule. Address emergency overcrawling with 429 or 503, and persistent URL problems through design.

Misconception

Repeated sitemap submission speeds up fetching

There is no need to submit the same unchanged sitemap many times a day. A sitemap is a hint and does not guarantee immediate fetching of every listed URL.

Misconception

All 4xx responses waste crawl budget

Google says 4xx responses other than 429 do not themselves waste crawl budget. Return a correct 404/410 for a deleted URL with no replacement.

Frequently asked questions about crawl budget

Do small sites need crawl budget optimization?

Usually no dedicated measure is needed. If pages are fetched within a reasonable time after publication, routinely review internal links, an up-to-date sitemap, canonicals, robots.txt mistakes, Page Indexing, and content quality.

Will more crawling improve search rankings?

Not necessarily. Crawling is not itself a ranking signal; rendering, assessment, duplicate consolidation, and indexing decisions follow it. Do not promise ranking gains merely by fixing missed important URLs.

Does noindex save crawl budget?

It is not a direct crawl-stop mechanism: Googlebot must fetch the page to read noindex. Use it to exclude pages from search, and manage generation, internal links, canonical signals, and robots.txt for URLs that need not be crawled.

Does blocking a URL in robots.txt remove it from search results?

Not necessarily. robots.txt controls crawling. If the URL is known through other pages, it may appear without its content being fetched. To exclude it, allow crawling so Google can recognize noindex.

Should every 404 be redirected?

No. Use 301 or 308 for a move with a replacement, and 404 or 410 for permanent deletion without one. Redirecting everything to an unrelated homepage can be treated as a soft 404.

Will a faster server always increase crawling?

No. Faster responses may relieve capacity constraints, but low demand can still keep crawling low. Look for evidence of hostload or availability constraints before scaling.

Can Search Console show a complete crawl history for a specific URL?

No. Crawl Stats shows overall trends and example URLs. Use server logs with verified Googlebot requests to establish fetch times and frequency for specific URLs or directories.

Does Google use sitemap priority and changefreq?

Google ignores priority and changefreq. List canonical URLs you want shown in search, and update lastmod accurately for significant content, structured data, or major link changes.

Official Google sources and verification date

The following primary sources were checked for specifications and decision criteria on July 29, 2026. Google's guidance changes; consult the latest version before making implementation changes.

Conclusion: use evidence to remove barriers to fetching important URLs

Crawl budget is not a number every site needs to increase. First assess applicability based on size, update frequency, the share of Discovered - currently not indexed URLs, and delays fetching important URLs.

If applicable, separate crawl capacity from demand and compare Crawl Stats, verified Googlebot logs, URL Inspection, and Page Indexing. Fix availability and configuration first; organize duplicate and faceted URLs, soft 404s, and redirect chains; and include canonical URLs and accurate lastmod in the sitemap.

Whether important URLs were fetched when needed, rather than total crawls, is the most practical success criterion for managing crawl budget.

Written and reviewed by: Finite Field Editorial Team We reviewed Google's official crawling infrastructure, Search Console, sitemap, canonical, robots.txt, noindex, and faceted navigation documentation and separated applicability, diagnosis, improvement, and measurement. We make no guarantee of rankings or indexing results.

Website Production & Technical SEO

Discuss website production and improvement with Finite Field

We design websites around information architecture, SEO, page speed, ease of updates, and inquiry paths as well as visual appearance. We handle WordPress redesigns and fast websites and business systems developed in Go.

Free consultation Starting with requirements WordPress and Go
  • Redesign page structure around search intent and business goals
  • Improve mobile presentation, page speed, and structured data together
  • Discuss ongoing operations, measurement, and integration with business systems
Contact

Tell us what is difficult about your current site

We can start by clarifying the problem even if you are unsure what to fix first.

We will review your request and clearly explain the scope and next steps.