Skip to content
SEO2026

Robots Exclusion Protocol · RFC 9309 · Google Search

Robots.txt Tester: Check Googlebot Allow, Disallow and URL Access

Test whether your pasted robots.txt rules allow Googlebot and other crawlers to access individual URLs. Select a crawler and see the matching user-agent group, winning rule and line number. Everything runs locally in your browser.

Free browser-local checker

Check robots.txt rules for your URLs

This does not contact your website; it tests only the rules you paste. Never paste secrets.

Preparing the checker…

How to test robots.txt correctly

1. Find the correct live robots.txt policy

Typically, the file lives at https://example.com/robots.txt. Policies apply to a particular scheme, host and port; rules from another subdomain do not govern this origin. This local workbench does not automatically fetch the live file, so make sure the text you paste reflects the policy you intend to assess.

2. Select the appropriate crawler

A specific Googlebot group takes precedence over the generic User-agent: * group. Multiple matching groups for the same product token are combined. Specialized crawlers can have different rules. Selecting Googlebot-Image does not imply the same policies apply to every search or AI crawler.

3. Apply Allow, Disallow, * and $ correctly

Disallow: /private/ blocks crawling of matching paths, but a longer Allow: /private/public-guide can open an exception. An asterisk matches arbitrary characters; a final dollar sign anchors the end. When two matching Google rules are equally specific, the least restrictive Allow wins.

4. Crawling is not indexing

Blocking a path in robots.txt does not reliably remove its URL from Google's index: links elsewhere can still expose the address. For index control, use supported noindex instructions in accessible HTML or HTTP response headers. If crawling is blocked, the crawler may never see the page's meta robots directives.

5. Verify the live site before releasing changes

Test critical category and product pages, resources, PDF files, query parameters and exceptions. Then check the actual robots.txt HTTP response, Google Search Console URL Inspection, server logs, canonical tags and rendered HTML. This simulator cannot reproduce downtime, caching, file retrieval failures or crawler-specific overrides.

6. Work with SEO and AI crawler policies responsibly

Googlebot and other search or AI bots may identify using different tokens. A result for one token should not be generalized to all. Version control rules, review crawl coverage after deployment, and never use robots.txt to store confidential paths. Sitemap hints do not override Disallow.

Frequently asked questions

Does Allow guarantee Google indexing?

No. Allow only means the modeled robots.txt policy does not block crawling. Indexing depends on content, accessibility, canonicalization and many other factors.

Why didn't Disallow: / block the URL?

A more specific Allow might apply, or a different user-agent group might have taken precedence.

Does Noindex: in robots.txt work?

Google does not support a Noindex: robots.txt directive for indexing control. Use a readable robots meta tag or X-Robots-Tag instead.

What does Sitemap do in robots.txt?

It identifies a sitemap location but does not override Disallow or grant crawl permission.

Methodology and official sources

Only locally pasted rules are analyzed: matching crawler group, longest applicable Allow or Disallow, outcome and source line. References:

Report factual errors to the author

Continue your SEO diagnosis

After robots.txt, verify Google indexing, HTML rendering, canonicalization, internal links and Search Console measurements.