Skip to content
SEO2026

Part II · HOW A SEARCH ENGINE SEES THE WEB

Crawling: queues, frequency, resources and priorities

Chapter 6 / 43 · 4,906 words · ~26 min read

Adapted directly from SEO 2026: Search in the AI Era by Mykhailo Oplakanets (Werter). Official author publication.

Читати українською

What crawl budget actually means, when it becomes a problem, how Googlebot responds to the server, what Crawl Stats shows and how to move from Search Console aggregates to event-level log analysis.

Data and sources as of: September 12, 2026

Crawl budget is often discussed as though every site has a fixed daily Googlebot allowance that can be “spent” on good or bad pages. This is a convenient metaphor, but technically inaccurate. In Google's documentation dated July 22, 2026, the model is described differently: the actual volume of crawling is determined by the intersection of two things - the host's crawl capacity limit and crawl demand for specific URLs.[1]

The first answers the question “how much can Google fetch without causing problems for the server”. The second answers “how much does Google actually want to fetch”. If the server is fast and stable but demand is low, Google is under no obligation to increase crawl frequency. If demand is high but the host starts responding more slowly or returning 5xx or 429, the system reduces the load. Crawl budget is therefore neither a single control nor a single counter.

Fig. 6.1. Google's crawl budget model: actual crawling emerges at the intersection of host capacity and crawl demand.
Fig. 6.1. Google's crawl budget model: actual crawling emerges at the intersection of host capacity and crawl demand.

Author's diagram based on Google Crawling Infrastructure. This is a conceptual model, not a formula for multiplying two metrics.

6.1. When crawl budget warrants specific attention at all

For most small sites, crawl budget is not a separate SEO problem. Google explicitly states that if there are few pages, they do not change very quickly or new content is crawled the same day, maintaining a sitemap and regularly checking the Page Indexing report is enough. Google's separate advanced guide is aimed primarily at large and highly dynamic sites.[1]

The official guidelines are approximate, not hard thresholds: 1 million or more unique URLs with moderate changes roughly once a week; 10 thousand or more unique URLs with very rapid daily changes; or sites where a substantial proportion of URLs in Search Console have the status “Discovered - currently not indexed”.[1] Google itself specifically cautions that these numbers are intended for classification, not for a mathematical decision of “9,999 URLs - no problem, 10,000 - a problem”.

Fig. 6.2. Google's guidelines for sites where crawl budget optimisation becomes a distinct technical task.
Fig. 6.2. Google's guidelines for sites where crawl budget optimisation becomes a distinct technical task.

Source: Google Crawling Infrastructure, 2026. Google describes both thresholds as approximate.

Table 6.1. Crawl budget is not a universal cause of slow indexing. First, we determine whether the site's scale and rate of change make it relevant at all.
SituationShould I start with crawl budgetWhat to check first
800 static pages, new URLs are crawled quicklyUsually notPage Indexing report, sitemap, internal linking, technical accessibility.
50 thousand products, prices and availability change dailyYes, it already makes senseLogs, Crawl Stats, recrawl frequency, 304, response time, unnecessary parameterised URLs.
2 million catalogue URLs, some templates barely changeYesURL inventory, duplicates, facets, soft 404s, recrawl demand.
20 thousand pages, but thousands are “Discovered - currently not indexed”Yes, as one hypothesisWhether the problem lies in new URL discovery, demand, quality, URL inventory or host capacity.
A site with 5 thousand URLs after a migrationMay be temporarily importantRedirects, sitemap, old/new URLs, status codes, logs and the recrawl wave.

6.2. Capacity limit: Google looks beyond request counts

Google determines the crawl capacity limit through the load on the host. It considers not only the number of parallel connections, but also how long the server keeps them open. If response time and latency are stable or improving, the limit may increase. If latency or TTFB deteriorates, or 5xx or 429 responses appear, the limit decreases.[1]

This is one reason why I do not use “requests per second” as a standalone KPI. Two servers can handle the same 5 requests per second in completely different ways. One serves HTML in 80 ms and barely notices Googlebot. The other keeps the connection open for 2 seconds, hits CPU or database bottlenecks and starts to degrade. To a crawler, these are different hosts, even if the request count is the same.

Table 6.2. The HTTP response affects not only indexing, but also the economics of recrawling.
Host-side signalWhat Google seesExpected response
Stable or improving response timeThe host is handling the loadThe capacity limit may increase if there is demand.
Latency / TTFB increasesEach connection takes more timeGoogle may reduce crawl intensity.
429 Too Many RequestsThe server explicitly signals overloadGoogle slows down crawling.
5xx across a substantial number of URLsServer error / unavailabilityGoogle temporarily reduces crawl frequency.
404 / 410 for deleted URLsContent is permanently absentThe frequency of repeat requests gradually falls; this is not rate limiting.
304 Not ModifiedContent has not changedGoogle can reuse the previous version without transferring the content again.

6.3. Crawl demand: a fast server does not mean Google wants more URLs

The other side of the model is crawl demand. For Googlebot, it depends on site size, update frequency, and the quality and relevance of pages relative to other sites. Google's current documentation specifically identifies perceived URL inventory, popularity and staleness. The factor most directly under the owner's control is URL inventory: if the system knows about millions of duplicates, sorting variants, obsolete addresses or other URLs that are not needed in search, some resources go towards them.[1]

Popularity here should not be reduced to the simplistic formula “more backlinks = more crawling”. Google phrases it more cautiously: popular URLs on the internet tend to be crawled more frequently to keep them fresh. Staleness means the system tries to avoid keeping outdated documents. A site migration or another site-wide event may also temporarily increase demand for recrawling.[1]

Table 6.3. Crawl demand is not a queue whose positions can be set manually. We control inventory and signals, not Google's internal scheduler.
Demand FactorWhat it means in practiceWhat SEO / development can do
URL inventoryHow many addresses Google knows about and considers potential candidatesEliminate infinite URL spaces, consolidate duplicates, clean up sitemaps, and return 404/410 where URLs have genuinely been removed.
Change frequencyHow quickly a document becomes outdatedAccurate lastmod, caching, dedicated monitoring of dynamic templates.
Quality and relevanceHow worthwhile it is for Google's product to reprocess the pagesDo not scale low-value URLs just because it is technically possible to generate them.
PopularityHow visible a URL is on the webDo not use it as an artificial crawl budget KPI; analyse it as part of the overall value of pages.
Site eventsMigration, bulk changes to URLs or contentExpect a temporary wave of recrawling and monitor server capacity.

6.4. Crawl Stats: what Search Console actually shows

The Crawl Stats report is the best starting point for a quick macro-level analysis. It shows total crawl requests, total download size, average response time, host status, and breakdowns by HTTP response, file type, crawl purpose and Googlebot type.[2] For a domain property, you can also view hosts, but the interface shows only 20 child domains that received crawl traffic over the last 90 days.[2]

The report has an important limitation: the list of example URLs is not a complete log. The absence of a particular URL from the examples does not prove that Google did not request it. Google itself describes these URLs as a representative sample. Crawl Stats therefore answers “what is happening to the host overall” well, but is less effective at answering “what happened to each URL”.[2]

Fig. 6.3. The breakdowns available in Crawl Stats in Search Console. No values have been invented for the diagram.
Fig. 6.3. The breakdowns available in Crawl Stats in Search Console. No values have been invented for the diagram.

A schematic representation of the official report. Search Console shows crawl purpose separately: discovery of a URL that Google has not crawled before, and refresh - recrawling a known URL.

Table 6.4. Search Console provides an aggregate view of system health; a server log is needed for the event level.
Crawl Stats fieldWhat can be establishedWhat can't be established
Total crawl requestsThe volume of Google's requests to URLs and resources on your hostHow many unique pages were fully reassessed.
Total download sizeHow many bytes Google downloaded during the periodThat a larger volume is a problem in itself.
Average response timeThe average response time for requested resourcesThe TTFB of a specific important template without segmentation.
Host statusWhether there were significant robots.txt, DNS or network connectivity issuesThe root cause of a business downturn without additional diagnostics.
Crawl purposeDiscovery or RefreshThe full priority of a URL within the scheduler.
Googlebot typeSmartphone, Desktop, Image, Video, page resources, etc.That every request had the same SEO value.
Example URLsExamples of requests in the selected segmentA complete list of all Googlebot requests.

6.5. Discovery and Refresh: discovering new URLs and recrawling

In Crawl Stats, Google divides crawl purpose into Discovery and Refresh. Discovery means that the URL has not previously been crawled by Google. Refresh is a repeat request for a known page.[2] This is a useful distinction for releases. If you have just published a large set of new pages or submitted a sitemap, it is reasonable to expect an increase in crawling of new URLs. If the problem is that prices, product availability or news are becoming outdated in search results, you need to look not only at the discovery of new URLs but also at recrawl frequency.

I do not set a KPI of “Discovery must account for X% of all requests”. Such a percentage is meaningless without context. The normal profile will differ for a news site, a catalogue after a large import and a mature encyclopaedic site. The right question is whether crawl frequency matches the business's rate of change for a particular page type.

6.6. Server logs: when aggregates are no longer enough

As soon as the question becomes “exactly which URLs does Googlebot crawl, how often and with which status codes?”, I turn to server logs. They contain what Search Console does not: the exact time, the specific URL, the HTTP status code, the response size, User-Agent, IP and, if the infrastructure logs it, response time. This allows requests to be grouped by patterns, parameters, status codes and time.

User-Agent is not enough for verification. Google explicitly warns that it can be spoofed. For a high-stakes audit, you need to verify IPs through reverse DNS lookups or match them against the official Googlebot IP ranges.[3] Later in this book, I use the term “Googlebot in the logs” only for traffic that has already passed this verification.

Fig. 6.4. A practical workflow from raw server logs to an SEO decision.
Fig. 6.4. A practical workflow from raw server logs to an SEO decision.

An unclassified log gives you millions of rows. Value emerges after bot verification, URL normalisation and mapping to page types.

Table 6.5. The minimum set of fields for analysing crawling from server logs.
Log fieldWhy it is SEOCommon mistake
Time (timestamp)Calculate first crawl times, recrawl intervals, peaks and delaysMix time zones or round to the day too early.
URL / path / queryBuild patterns and identify parameter spacesAnalyse each URL individually and lose sight of the structure.
HTTP status code (status)Separate 2xx, redirects, 404/410, 429, 5xxLook only at the number of 200 responses.
Response size (bytes sent)Assess network traffic and large responsesConfuse compressed and uncompressed bytes from different logging layers.
response timeIdentify degradation specifically in Googlebot requestsCompare different servers without normalisation.
User-Agent + IPVerify the crawler typeTrust only the Googlebot string in the UA.
Referral source (referrer)Sometimes useful for other bots/requestsExpect Googlebot always to pass the source where it discovered new URLs.

6.7. Metrics I calculate from logs

There is no universal “correct crawl frequency”, so I do not compare a site against an abstract industry percentage. I establish a baseline by page type, then examine how it changes after releases, migrations, facet restructuring or server optimisation.

Formula 6.1. Useful crawl share.

This is an internal operational metric. A “target canonical” is defined by your inventory, not by an external standard.

Formula 6.2. Unwanted crawl share.

Non-target URLs can include facets, internal search, obsolete parameters, infinite calendars and other URLs that you deliberately do not want to maintain as pages for search.

Formula 6.3. First crawl latency.

Calculate the median and percentiles separately by page type. A few extreme long-tail values can easily distort the mean.

Formula 6.4. Recrawl interval.

Aggregate by page type, rather than using a single number for the entire site.

Table 6.6. Log metrics must be tied to a URL type and a specific technical hypothesis.
MetricLevel of analysisWhat I want to see after the fix
Useful crawl shareHost → directory → templateA larger share of requests goes to target URLs if we have genuinely removed wasteful URL spaces.
Unwanted crawl shareParameter / facet / internal searchA decline in requests to URLs with no search value.
First crawl latencyNew URLs of the same typeA shorter median time from publication to the first verified crawl.
Median recrawl intervalFrequently changing URLsFrequency better matches the actual rate of change.
5xx/429 shareHost and individual templatesFewer errors and no correlation between crawl peaks and server degradation.
304 shareStable URLsCorrect freshness validation for unchanged documents without unnecessary content transfer.

6.8. 304 Not Modified: savings without an imaginary “ranking boost”

Google explicitly recommends supporting HTTP caching and 304 Not Modified. If a document has not changed since the previous crawl, a 304 tells the crawler that it can use the cached version without transferring the full content again. This saves network traffic and server resources.[1][5]

At the same time, 304 should not be sold as a “ranking factor” or a way to force Google to crawl more. The correct conclusion is more modest: for large sites with a high share of repeat requests for unchanged content, freshness validation can make repeat retrieval cheaper. Whether those savings translate into more crawling of your important URLs depends on demand and capacity, not on a single header.

6.9. Faceted navigation: the most common way to create more URLs than the business needs

Google identifies faceted navigation as one of the most frequent causes of excessive crawling that site owners bring to the Search team. The problem is not filters themselves, but the combinatorial URL space: each attribute and each ordering of parameters can generate a new URL that the crawler cannot assess before requesting it.[4]

If these pages are not needed in search, Google recommends preventing them from being crawled, for example through robots.txt, or designing filters so that they do not create crawlable URLs. If some facets are needed as landing pages, the task is more complex: you need to clearly separate combinations that should be indexed from the technical “long tail” of parameters, maintain a stable parameter order, return 404 for empty combinations and avoid creating infinite spaces.[4]

Table 6.7. Excessive crawling almost always starts with an uncontrolled URL inventory, not an “overactive Googlebot”.
PatternWhy it's expensiveWhat to check
Facets and sortingThe number of combinations grows faster than the useful inventoryWhich combinations have search demand; whether the rest are crawlable; parameter order; canonical; robots.
Internal searchEach query can create a new URLWhether there are crawlable links to search results; whether these URLs should be in the index.
CalendarsCan generate endless navigation into the future/pastDate boundaries, nofollow/robots, 404 for non-existent intervals.
Session IDs / tracking parametersOne document acquires many technical URLsParameter sources, internal links, canonical, server-side normalisation.
Protocol/host/path duplicatesInventory multiplies without adding valueRedirects, canonical, internal URLs, sitemap, host configuration.
Soft 404The URL returns 200 but actually has no useful contentA proper 404/410 or real content instead of an empty template page.

6.10. HTTP status codes: do not try to control crawl rate with the wrong codes

Google explicitly warns against using 401, 403 or 404 for rate limiting. No 4xx code other than 429 signals “the server is overloaded”. 429 specifically means Too Many Requests and is treated as a server overload signal. 5xx codes also reduce crawling intensity.[5]

In an emergency where crawling traffic needs to be reduced sharply for a few hours or one or two days, Google recommends returning 500, 503 or 429. This is a blunt instrument: it affects the entire host, slows the discovery of new URLs and recrawling, and if the errors persist for several days, URLs may start dropping out of the index. It is not a permanent mechanism for “budget optimisation”.[5]

Fig. 6.5. The most important HTTP responses from a crawling perspective.
Fig. 6.5. The most important HTTP responses from a crawling perspective.

The diagram summarises Google’s official documentation. For full details on how each status code is handled, see source 5.

6.11. robots.txt - critical infrastructure, not a file you can serve “however it happens to work”

robots.txt itself also needs to be crawled. Google usually caches it for up to 24 hours, but may retain the cache longer if an update is unavailable. A particularly dangerous scenario is a 5xx response for robots.txt. The official specification describes the behaviour over time: for the first 12 hours, Google stops crawling the site but continues trying to fetch robots.txt; then, for up to 30 days, it may use the last available version, if one exists; after 30 days, its behaviour depends on the site’s overall availability.[5]

Fig. 6.6. A simplified timeline of Google’s behaviour when robots.txt returns 5xx.
Fig. 6.6. A simplified timeline of Google’s behaviour when robots.txt returns 5xx.

Source: Google Crawling Infrastructure. The diagram does not replace the full specification for DNS, timeouts and other network errors.

The practical takeaway: robots.txt availability needs to be monitored as an availability-critical endpoint. If a CDN, WAF or deployment accidentally starts returning 5xx for /robots.txt, the consequences are not limited to the file itself. This can stop or substantially alter crawling across the entire host.

6.12. The byte limit: “the page opens in a browser” does not mean Googlebot fetched all of it

In March 2026, Google provided a separate explanation of its fetching infrastructure limits. For Googlebot’s general web search crawling, up to 2 MB of a single supported resource is fetched in uncompressed form, including HTTP headers; for PDFs, the limit is 64 MB. Once the limit is reached, the crawler stops fetching and passes on only the portion already received. Each CSS or JavaScript resource referenced by the HTML is requested separately and has its own limit.[5]

Fig. 6.7. Official size limits for a single Googlebot fetch in 2026.
Fig. 6.7. Official size limits for a single Googlebot fetch in 2026.

2 MB for supported file types in general web search; 64 MB for PDFs. Other Google crawlers may have different limits.

For most HTML pages, 2 MB of uncompressed HTML is a very large amount. But with massive templates used for page generation at scale, long catalogues or pages with embedded data, this is no longer a theoretical limit. I check not only the transfer size after gzip/brotli compression, but also the actual uncompressed response body, because that is what Google’s stated limit applies to.

6.13. Python: minimal analysis of verified Googlebot logs

Below, I deliberately avoid trying to parse every possible nginx/Apache format. Different teams already send logs to S3, BigQuery, ClickHouse or CSV with different fields. The script expects a normalised CSV after Googlebot verification, with timestamp, url, status, bytes and response_time columns. The SEO task then is to turn requests into understandable patterns.

NO CODE: PROMPT FOR AI

I will upload verified_googlebot.csv with timestamp, url, status, response_time and bytes columns. Check data types and missing values. Segment URLs using the rules I provide for product/category/search/other, calculate the number of requests and unique URLs, median response time, volume of bytes transferred, proportions of 2xx/3xx/4xx/5xx responses and median recrawl interval. Show anomalies, but do not draw causal conclusions without additional data.

ALTERNATIVE WITHOUT AI

If the CSV is already normalised, the same can be done in Excel/Sheets: add a page-type column and a status-code class, then create a Pivot Table. For the recrawl interval, sort by URL + timestamp and calculate the difference from the previous request; Power Query is more convenient for large files.

QR: CODE

Scan with your camera: the QR code contains a compact, executable version of the code without comments. The commented code is provided below.

FOR AUTOMATION: PYTHON FOR LOG ANALYSIS

Code 6.1. Basic segmentation of verified Googlebot requests by page type, status code and median recrawl interval.

PythonOriginal book example
import pandas as pd
from urllib.parse import urlsplit

# Expected fields: timestamp, url, status, response_time, bytes.
# The CSV should contain already verified Googlebot requests.
df = pd.read_csv(
    "verified_googlebot.csv",
    parse_dates=["timestamp"]
)

# 1) Determine the page type by URL path.
# Replace the rules below with the structure of your site.
def page_type(url):
    path = urlsplit(url).path
    if path.startswith("/product/"):
        return "product"
    if path.startswith("/category/"):
        return "category"
    if path.startswith("/search/"):
        return "internal_search"
    return "other"

# 2) Add the page type and HTTP status family.
df["page_type"] = df["url"].map(page_type)
df["status_family"] = (df["status"] // 100).astype(str) + "xx"

# 3) Summary: how many requests, how many URLs,
# median response time and bytes transferred.
summary = (
    df.groupby("page_type")
      .agg(
          requests=("url", "size"),
          unique_urls=("url", "nunique"),
          median_response_time=("response_time", "median"),
          downloaded_bytes=("bytes", "sum"),
      )
      .sort_values("requests", ascending=False)
)

# 4) Proportion of 2xx/3xx/4xx/5xx within each page type.
status = pd.crosstab(
    df["page_type"],
    df["status_family"],
    normalize="index"
)

# 5) Median interval between re-requests to the same URL.
ordered = df.sort_values(["url", "timestamp"])
ordered["prev"] = ordered.groupby("url")["timestamp"].shift(1)
ordered["recrawl_hours"] = (
    ordered["timestamp"] - ordered["prev"]
).dt.total_seconds() / 3600
recrawl = ordered.groupby("page_type")["recrawl_hours"].median()

print(summary)
print(status)
print(recrawl)

After this, I add business fields: target canonical URL, indexability, template version, last-modified time, publication date and inventory status. The log then stops being a “technical file” and becomes a system that can answer questions such as: how many hours, at the median, Googlebot needs to crawl a new product for the first time; what percentage of its requests go to /search/; whether the proportion of 5xx responses fell after an infrastructure change; whether the recrawl interval shortened for pages where prices change daily.

6.14. Diagnostics: what to do if Google crawls “the wrong things”

Table 6.8. A crawling symptom does not equal a cause. Each hypothesis requires a separate data source.
SymptomStrongest first hypothesisData to testAction
Many crawl requests to facetsURL inventory bloated by parametersLogs + site crawl + parameters + robotsDefine the set of facets that should be indexed; remove the rest from the crawlable graph or block them according to the strategy.
New URLs wait a long time for their first crawlLow demand, weak discovery of new URLs or an overloaded hostPublication time, first Googlebot request, sitemap, internal links, Crawl StatsSeparate new URL discovery issues from server capacity issues; do not treat everything with the request indexing button.
Crawl spikes coincide with 5xx responsesThe host cannot handle the loadResponse time, 5xx/429 by minute, CPU/DB/CDN metricsIncrease server capacity; do not block Googlebot with 403/404 responses.
Google constantly crawls deleted URLsOld inventory continues to receive signals or incorrect responsesStatus-code history, internal/external links, sitemap, redirectsReturn 404/410 for genuinely deleted pages; remove links and URLs from the sitemap; avoid soft 404s.
Stable pages return a full 200 response on every recrawlNo effective freshness validationETag/Last-Modified/If-Modified-Since, logsConfigure correct HTTP caching and 304 responses where the document genuinely has not changed.
Crawl Stats shows slower responsesThe infrastructure or template has become more resource-intensiveBreakdown by file type / host + server metricsIdentify the specific resource type or endpoint; do not draw conclusions from the site-wide average alone.

6.15. The crawl budget audit sequence I use

I start with the inventory, not the log. If the team itself does not know which URL types should exist in search, it is impossible to determine what is “wasteful” in crawl traffic. Next, I look at Crawl Stats to understand the scale, host status, response-time profile and discovery/refresh ratio. Only then do I move on to logs, because without context, millions of rows can easily become attractive but meaningless charts.

Table 6.9. A crawling audit as a sequence of checks, not a list of 100 technical checkboxes.
StageQuestionOutput Artifact
1. URL inventoryWhich URLs does the business want to maintain as pages for search?Matrix: page type → intended for indexing / not intended for indexing / crawlable / blocked.
2. ArchitectureWhere do unnecessary URLs come from?Map of parameters, facets, sort orders, internal search and calendar URL spaces.
3. Crawl StatsIs there a scale or host health issue?Trends in requests, bytes, response time, 5xx/429, Discovery/Refresh.
4. LogsWhich URLs does Googlebot actually request?Distribution of requests by template, status code, interval and delay to first crawl.
5. ServerIs Googlebot hitting server capacity limits?Correlation of crawl traffic with latency, 5xx/429, CPU, DB, CDN.
6. HypothesisWhat exactly are we changing, and what effect do we expect?Before/after comparison using a specific metric, not “Googlebot got better”.
7. MonitoringHave we made new URL discovery / deindexing / UX worse?Parallel monitoring: Page Indexing report, GSC, sitemap and organic visibility.

6.16. What you definitely should not do

Do not block everything indiscriminately in robots.txt just because Googlebot visits it frequently. If a URL first needs to be removed from the index using noindex, the crawler must be able to retrieve that directive. Do not make random daily changes to robots.txt “to redistribute the budget”: Google explicitly advises against using the file as a temporary redistribution mechanism. Do not return 403 or 404 to “slow the bot down”. And do not measure success simply by a drop in the total number of crawl requests - fewer requests can mean either greater efficiency or a loss of demand.[1][5]

6.17. Myths I would throw out of technical SEO

Table 6.10. Crawl budget without the mythology: what actually follows from Google’s current documentation.
MythWhat is provenPractical conclusion
“Every site has a fixed daily crawl budget”Google describes an adaptive model of capacity and demand.Do not look for a single number that Search Console does not show anyway.
“Every site needs crawl budget optimisation”Google specifically targets its advanced guide at large/dynamic sites or sites with a substantial proportion of URLs marked “Discovered - currently not indexed”.For a small site, new URL discovery, indexability, canonicals and quality are more often what matters.
“A faster server automatically means more crawling”Better server capacity can raise the crawl capacity limit, but crawl demand is also needed.Server optimisation is necessary when the server is the bottleneck, but it does not create demand on its own.
“404 saves crawl budget by limiting the crawl rate”4xx responses, except 429, do not reduce the crawl rate as an overload signal.Use 404/410 to indicate removed content, not for rate limiting.
“Crawl Stats is a complete Googlebot log”The URL examples in the report are representative, but not exhaustive.URL-level analysis requires server logs.
“noindex saves crawling”Google must fetch the page to read noindex.Do not confuse indexing control with crawling control.
“A drop in crawl requests after changes is always good”Fewer requests can also mean lower demand.Assess crawl delays for important URLs, coverage and recrawling, rather than the total request count in isolation.

6.18. What we know for certain and what remains a hypothesis

We know for certain: Google defines crawl budget through capacity and demand; a large, unnecessary URL inventory can reduce efficiency; response time, 5xx and 429 affect crawl capacity; popularity and staleness affect demand; 304 can save network traffic; Search Console Crawl Stats is aggregated, and its URL examples are not a complete log; faceted navigation can create a practically infinite URL space; Googlebot has documented byte limits per fetch.[1][2][4][5]

What is not publicly known: the scheduler’s exact formula, the weight of each signal, the individual “cost” of a particular URL in the crawl queue, or the guaranteed amount of additional crawling after speeding up the server or blocking facets. If an SEO specialist claims “Google allocated us 187,432 notional crawl credits per day” without internal Google data, that is not a fact but a model or a fabrication.

6.19. Practical takeaway

Crawl budget only makes sense when we stop treating it as a magical limit. A large site has a URL inventory, server capacity, search engine demand, a history of changes and specific HTTP responses. SEO can have a very strong influence on the first, a partial influence on the second, an indirect influence on the third, and direct control over the accuracy of the fourth and fifth.

My working criterion is simple: important new URLs should receive their first crawl quickly, important updated URLs should be recrawled at a frequency that matches their business dynamics, and the server should not spend a substantial share of its resources on URLs that the product does not intend to maintain in search. Everything else is a means of measuring this system.

Sources for Chapter 6

1. Google Crawling Infrastructure: Optimise your crawl budget. https://developers.google.com/crawling/docs/crawl-budget Updated July 22, 2026.

2. Google Search Console Help: Crawl Stats report. https://support.google.com/webmasters/answer/9679690?hl=en

3. Google Search Central: Googlebot. https://developers.google.com/search/docs/crawling-indexing/googlebot Updated February 3, 2026.

4. Google Crawling Infrastructure: Managing crawling of faceted navigation URLs. https://developers.google.com/crawling/docs/faceted-navigation

5. Google Search Central / Crawling Infrastructure: technical crawling references. https://developers.google.com/search/blog/2026/03/crawler-blog-post ; https://developers.google.com/crawling/docs/crawlers-fetchers/reduce-crawl-rate ; https://developers.google.com/crawling/docs/troubleshooting/http-status-codes ; https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec Googlebot fetch limits, emergency crawl rate reduction, HTTP status codes and robots.txt behaviour.

Practical tools for this chapter

Apply the book's methods with a reusable checklist, a URL diagnostic worksheet or measurable evidence. Each field guide includes sources and acceptance tests.

All SEO practice guides

Contextual reference · SEO 2026

Key terms related to this chapter

Editorial explanations of related chapter concepts with examples, common mistakes and primary sources. This reference does not modify the book's original text.

Explore all 29 SEO glossary definitions

Original book illustrations appear next to their corresponding captions when the source media is available.