Robots.txt Tester & Crawl Rule Checker
Fetch a live robots.txt from any website — or paste your own — and test whether a specific URL can be crawled. This free tool parses User-agent groups, Allow and Disallow rules and Sitemap lines exactly like Google does: longest-match wins, with full wildcard (*) and end-anchor ($) support. Built for SEOs, developers and site owners who need a clear Allowed or Blocked verdict.
Test a URL
Enter a path on this site, or a full URL on the same host.
| Matching rule | |
|---|---|
| Matched group | |
| Tested path (encoded) Google matches rules against the percent-encoded path, so we test that form. | |
Parse warnings
User-agent groups
| User-agents | Count |
|---|
Sitemaps
No Sitemap: lines declared in this file.
robots.txt contents
Test any site's robots.txt
Fetch a live robots.txt or paste one, then check whether a URL is crawlable by Googlebot and other bots.
How to use
- Enter a site URL and click Fetch — we build {origin}/robots.txt for you. Or paste an existing file.
- Review the raw file with line numbers, parse warnings, groups and declared sitemaps.
- Type the path or full URL you want to check and pick a crawler (default Googlebot).
- Click Test to see an Allowed or Blocked verdict with the exact rule line that matched.
- Copy the verdict summary to share with your team or paste into a ticket.
Features
- Fetches robots.txt server-side and shows the raw file with line numbers
- Google-accurate matching: longest rule wins, ties go to Allow, wildcards * and $ supported
- Test against Googlebot, Googlebot-Image, Googlebot-News, Bingbot, Twitterbot, facebookexternalhit or a custom agent
- Flags parse issues: unknown directives, rules before the first User-agent, and BOM problems
- Handles Thai and other UTF-8 paths by testing the percent-encoded form Google actually matches
- Explains informational states — missing (404) means allow-all, repeated 5xx means block-all
Frequently asked questions
What happens if a site has no robots.txt?
A 404 or missing file is not an error — crawlers treat it as permission to crawl the entire site. The tool shows this as an informational "allow all" state rather than a failure.
How does Allow vs Disallow get decided when both match?
Following Google's rules, the most specific rule wins — the one with the longest matching path. If an Allow and a Disallow rule match with equal length, Allow wins. The tool always shows you the exact line that decided the verdict.
Do you support wildcards and the $ end-of-path marker?
Yes. * matches any sequence of characters and $ anchors the rule to the end of the URL path, matching Google's and RFC 9309 semantics. This lets you test rules like Disallow: /*.pdf$ correctly.
Why do you test the percent-encoded version of a Thai URL?
Google compares robots.txt rules against the percent-encoded path, so a Thai URL is matched in its %E0%B8… form. The tool encodes your test path the same way and shows both the readable and encoded versions.
What does a 5xx response on robots.txt mean?
A single 5xx may be temporary, but Google treats a robots.txt that keeps returning 5xx as "disallow everything," which can deindex your whole site. The tool warns you when it sees a server error so you can fix it fast.