Skip to content
SEO2026

Part II · HOW A SEARCH ENGINE SEES THE WEB

Indexing: why crawling does not guarantee indexing

Chapter 9 / 43 · 1,255 words · ~7 min read

Adapted directly from SEO 2026: Search in the AI Era by Mykhailo Oplakanets (Werter). Official author publication.

Читати українською

How to distinguish “Googlebot visited the URL” from “the document is in the index”, why noindex, duplicates, soft 404s, quality and technical rules produce different classes of problems, and how to build an indexing audit without the “Request indexing” ritual.

Data and sources as of: September 14, 2026

Indexing is where SEO most often confuses fact with intent. Googlebot received a 200 - so the page must be in the index. The URL is in the sitemap - so it must be in the index. We clicked Request indexing - so the system is “obliged” to add it. None of these statements reflects how Google describes Search. Even full compliance with the technical requirements does not guarantee crawling, indexing or serving.[1]

Fig. 9.1. Crawling is a prerequisite for many indexing decisions, but not a guarantee of inclusion in the index.
Fig. 9.1. Crawling is a prerequisite for many indexing decisions, but not a guarantee of inclusion in the index.

Author’s diagram based on Google Search Central’s How Search Works and Crawling & Indexing documentation.

9.1. The index is not an archive of everything Googlebot has seen

There is no reason for a search engine to keep every technically accessible URL in its active index. Parameter variants, duplicates, empty filters, test pages, soft 404s, low-value combinations of programmatic templates and infinite calendars may be crawlable without providing distinct search value. “Crawled - currently not indexed” should therefore be treated as a status category, not as a single defect with a single fix.

9.2. noindex works only when the crawler can see it

Google explicitly states that noindex must be accessible to the crawler. If the page is also blocked in robots.txt, Googlebot does not fetch the HTML and cannot see the noindex; the URL may still be known through links and sometimes appear in results without a normal snippet.[2] This is a classic example of a conflict between crawl control and index control.

Table 9.1. Four different ways to “remove a URL” solve different problems.
MechanismWhat it actually doesWhen to use
robots.txtRestricts crawler requests to URLs; it is not a reliable way to remove them from the index.When you do not want to spend crawl resources on a URL space or on resources that do not need to be fetched.
noindexAfter crawling, instructs the system not to keep the document in Search results.For pages that are accessible to users but should not appear in search.
404/410Indicates that the resource does not exist. Over time, the URL drops out of the index.The page has actually been removed without a relevant replacement.
301/308Redirects the user and sends a strong canonical signal to the new URL.There is an equivalent new address or a migration.

9.3. “Discovered” and “Crawled” are different bottlenecks

If a URL has the status Discovered - currently not indexed, the first question is whether Googlebot has fetched it. If not, we are still at the discovery/crawl scheduling stage: internal linking, sitemaps, server capacity, crawl demand and excessive URL space. If the URL is Crawled - currently not indexed, the crawler has already reached the document, and blindly “strengthening the sitemap” makes less sense. You need to check the canonical cluster, soft 404s, duplicates, template quality, the uniqueness of factual data and whether the page warrants a separately indexed document at all.

9.4. A soft 404 is not just a page that says “not found”

A soft 404 occurs when the HTTP response is formally successful, but the content looks like a missing or unusable resource. Typical examples include an empty category returning 200, a product that is permanently unavailable with no alternative, an internal search page with no results, and a template with one heading and no main content. An important cause of the problem is that the system has to interpret the semantic state separately instead of receiving the correct HTTP status.

9.5. A duplicate may be “not indexed” because its representative is indexed

In Indexing reports, many URLs that SEOs describe as having “dropped out of the index” are actually alternate pages or duplicates. This is not necessarily a problem. If parameterised URLs, a print version and a clean URL form one canonical cluster, indexing one representative is the expected behaviour. Here, you need to evaluate the cluster rather than try to get every URL indexed individually.

9.6. Quality in indexing: do not turn the unknown into a “ranking factor”

Google does not publish its complete indexing selection formula. So I do not write “the page is not indexed because of E-E-A-T” or “you need 800 words”. More accurately: Search Essentials explicitly does not guarantee indexing even for a technically valid page; Google evaluates content and duplicates during processing; if entire classes of thin or nearly interchangeable pages consistently fail to get indexed, that is a strong signal to reconsider the value of the template itself rather than look for a magic technical flag.[1][3]

9.7. The indexing funnel: the right denominator

Table 9.2. The denominators needed for a sound indexing report.
MetricNumeratorCorrect denominator
Indexable coverageIndexable URLs that are in the indexAll URLs the business actually wants indexed
Crawl-to-index rateIndexed URLs from the crawled sampleCrawled indexable URLs from the same template/period
Sitemap index rateIndexed canonical URLs from the sitemapValid canonical URLs returning 200 in the sitemap
Template index rateIndexed URLs using the templateAll indexable URLs using this template

I deliberately do not divide “indexed URLs” by “all URLs the crawler found”. If the crawler found 12 million parameter URLs, while the business expects 400 thousand indexed pages, that denominator tells us nothing about indexing quality.

9.8. Sampling instead of manual URL Inspection

URL Inspection is useful for verifying the state of a specific page, but checking 100 thousand URLs manually is not feasible. I build a stratified sample: template, internal linking level, new/old URLs, sitemap/non-sitemap, canonical/non-canonical, country/language. Then I take 50-200 URLs from each meaningful segment. The goal is not to obtain an “overall indexing %”, but to find a class where the probability of indexing differs sharply.

9.9. Request indexing - a tool for checking and accelerating individual changes, not an API for bulk indexing

I use a manual indexing request for an important new URL or after fixing a specific technical error, when I want to trigger the next cycle sooner. This is not a strategy for scale. Google explicitly states that crawling and indexing take time and are not guaranteed; a sitemap and a sound internal link graph are systematic channels, while the manual button is a supporting tool.[4]

9.10. A practical diagnostic sequence

My sequence: 1) should the URL exist as a separate searchable document; 2) final HTTP status; 3) robots/noindex; 4) canonical and duplicates; 5) rendered content; 6) internal linking and sitemap; 7) URL Inspection; 8) server log; 9) comparison of the template with indexed equivalents. This is almost always faster than starting with “let’s add more text” or “let’s buy links so Google will want to index it”.

9.11. What we know for certain

Technical accessibility does not guarantee indexing; noindex must be accessible to the crawler; a sitemap helps discovery but does not guarantee indexing or ranking; canonicalisation may mean that an alternative URL is not represented separately; Google provides no timelines or guarantees for crawling/indexing.[1][2][4]

Sources for Chapter 9

1. Google Search Central. Search Essentials. https://developers.google.com/search/docs/essentials

2. Google Search Central. Block Search indexing with noindex. https://developers.google.com/search/docs/crawling-indexing/block-indexing

3. Google Search Central. In-depth guide to how Google Search works. https://developers.google.com/search/docs/fundamentals/how-search-works

4. Google Search Central. Crawling and indexing FAQ. https://developers.google.com/search/help/crawling-index-faq

5. Google Search Central. Crawling and indexing overview. https://developers.google.com/search/docs/crawling-indexing

Practical tools for this chapter

Apply the book's methods with a reusable checklist, a URL diagnostic worksheet or measurable evidence. Each field guide includes sources and acceptance tests.

All SEO practice guides

Contextual reference · SEO 2026

Key terms related to this chapter

Editorial explanations of related chapter concepts with examples, common mistakes and primary sources. This reference does not modify the book's original text.

Explore all 29 SEO glossary definitions

Original book illustrations appear next to their corresponding captions when the source media is available.