How to distinguish “Googlebot visited the URL” from “the document is in the index”, why noindex, duplicates, soft 404s, quality and technical rules produce different classes of problems, and how to build an indexing audit without the “Request indexing” ritual.
Data and sources as of: September 14, 2026
Indexing is where SEO most often confuses fact with intent. Googlebot received a 200 - so the page must be in the index. The URL is in the sitemap - so it must be in the index. We clicked Request indexing - so the system is “obliged” to add it. None of these statements reflects how Google describes Search. Even full compliance with the technical requirements does not guarantee crawling, indexing or serving.[1]
Author’s diagram based on Google Search Central’s How Search Works and Crawling & Indexing documentation.
9.1. The index is not an archive of everything Googlebot has seen
There is no reason for a search engine to keep every technically accessible URL in its active index. Parameter variants, duplicates, empty filters, test pages, soft 404s, low-value combinations of programmatic templates and infinite calendars may be crawlable without providing distinct search value. “Crawled - currently not indexed” should therefore be treated as a status category, not as a single defect with a single fix.
9.2. noindex works only when the crawler can see it
Google explicitly states that noindex must be accessible to the crawler. If the page is also blocked in robots.txt, Googlebot does not fetch the HTML and cannot see the noindex; the URL may still be known through links and sometimes appear in results without a normal snippet.[2] This is a classic example of a conflict between crawl control and index control.
| Mechanism | What it actually does | When to use |
|---|---|---|
| robots.txt | Restricts crawler requests to URLs; it is not a reliable way to remove them from the index. | When you do not want to spend crawl resources on a URL space or on resources that do not need to be fetched. |
| noindex | After crawling, instructs the system not to keep the document in Search results. | For pages that are accessible to users but should not appear in search. |
| 404/410 | Indicates that the resource does not exist. Over time, the URL drops out of the index. | The page has actually been removed without a relevant replacement. |
| 301/308 | Redirects the user and sends a strong canonical signal to the new URL. | There is an equivalent new address or a migration. |
9.3. “Discovered” and “Crawled” are different bottlenecks
If a URL has the status Discovered - currently not indexed, the first question is whether Googlebot has fetched it. If not, we are still at the discovery/crawl scheduling stage: internal linking, sitemaps, server capacity, crawl demand and excessive URL space. If the URL is Crawled - currently not indexed, the crawler has already reached the document, and blindly “strengthening the sitemap” makes less sense. You need to check the canonical cluster, soft 404s, duplicates, template quality, the uniqueness of factual data and whether the page warrants a separately indexed document at all.
9.4. A soft 404 is not just a page that says “not found”
A soft 404 occurs when the HTTP response is formally successful, but the content looks like a missing or unusable resource. Typical examples include an empty category returning 200, a product that is permanently unavailable with no alternative, an internal search page with no results, and a template with one heading and no main content. An important cause of the problem is that the system has to interpret the semantic state separately instead of receiving the correct HTTP status.
9.5. A duplicate may be “not indexed” because its representative is indexed
In Indexing reports, many URLs that SEOs describe as having “dropped out of the index” are actually alternate pages or duplicates. This is not necessarily a problem. If parameterised URLs, a print version and a clean URL form one canonical cluster, indexing one representative is the expected behaviour. Here, you need to evaluate the cluster rather than try to get every URL indexed individually.
9.6. Quality in indexing: do not turn the unknown into a “ranking factor”
Google does not publish its complete indexing selection formula. So I do not write “the page is not indexed because of E-E-A-T” or “you need 800 words”. More accurately: Search Essentials explicitly does not guarantee indexing even for a technically valid page; Google evaluates content and duplicates during processing; if entire classes of thin or nearly interchangeable pages consistently fail to get indexed, that is a strong signal to reconsider the value of the template itself rather than look for a magic technical flag.[1][3]
9.7. The indexing funnel: the right denominator
| Metric | Numerator | Correct denominator |
|---|---|---|
| Indexable coverage | Indexable URLs that are in the index | All URLs the business actually wants indexed |
| Crawl-to-index rate | Indexed URLs from the crawled sample | Crawled indexable URLs from the same template/period |
| Sitemap index rate | Indexed canonical URLs from the sitemap | Valid canonical URLs returning 200 in the sitemap |
| Template index rate | Indexed URLs using the template | All indexable URLs using this template |
I deliberately do not divide “indexed URLs” by “all URLs the crawler found”. If the crawler found 12 million parameter URLs, while the business expects 400 thousand indexed pages, that denominator tells us nothing about indexing quality.
9.8. Sampling instead of manual URL Inspection
URL Inspection is useful for verifying the state of a specific page, but checking 100 thousand URLs manually is not feasible. I build a stratified sample: template, internal linking level, new/old URLs, sitemap/non-sitemap, canonical/non-canonical, country/language. Then I take 50-200 URLs from each meaningful segment. The goal is not to obtain an “overall indexing %”, but to find a class where the probability of indexing differs sharply.
9.9. Request indexing - a tool for checking and accelerating individual changes, not an API for bulk indexing
I use a manual indexing request for an important new URL or after fixing a specific technical error, when I want to trigger the next cycle sooner. This is not a strategy for scale. Google explicitly states that crawling and indexing take time and are not guaranteed; a sitemap and a sound internal link graph are systematic channels, while the manual button is a supporting tool.[4]
9.10. A practical diagnostic sequence
My sequence: 1) should the URL exist as a separate searchable document; 2) final HTTP status; 3) robots/noindex; 4) canonical and duplicates; 5) rendered content; 6) internal linking and sitemap; 7) URL Inspection; 8) server log; 9) comparison of the template with indexed equivalents. This is almost always faster than starting with “let’s add more text” or “let’s buy links so Google will want to index it”.
9.11. What we know for certain
Technical accessibility does not guarantee indexing; noindex must be accessible to the crawler; a sitemap helps discovery but does not guarantee indexing or ranking; canonicalisation may mean that an alternative URL is not represented separately; Google provides no timelines or guarantees for crawling/indexing.[1][2][4]
Sources for Chapter 9
1. Google Search Central. Search Essentials. https://developers.google.com/search/docs/essentials
2. Google Search Central. Block Search indexing with noindex. https://developers.google.com/search/docs/crawling-indexing/block-indexing
3. Google Search Central. In-depth guide to how Google Search works. https://developers.google.com/search/docs/fundamentals/how-search-works
4. Google Search Central. Crawling and indexing FAQ. https://developers.google.com/search/help/crawling-index-faq
5. Google Search Central. Crawling and indexing overview. https://developers.google.com/search/docs/crawling-indexing