Robots.txt Generator
Create a correct robots.txt file without memorising the syntax. Choose a preset — allow all, block all, or block common admin paths — then add user-agent groups with their own Allow and Disallow rules, an optional crawl-delay, and any sitemap URLs. The file is built live as you type, ready to copy or download. Everything runs in your browser, so nothing you enter is ever sent to a server.
Quick presets
User-agent groups
Start from a preset, then fine-tune the rules below. Presets replace the current groups.
One full URL per line. These apply to all crawlers, regardless of the groups above.
Generated robots.txt
Save this as robots.txt in your website root, e.g. https://example.com/robots.txt
How to use
- Pick a preset, or start from the default "allow all" group.
- Set the user-agent (use * for all bots, or a name like Googlebot).
- Add Disallow paths to block and Allow paths to permit, one per line.
- Optionally set a crawl-delay and add your sitemap URLs.
- Copy the generated robots.txt or download it and upload it to your site root.
Features
- Presets for allow all, block all, and blocking common admin paths
- Multiple user-agent groups, each with its own Allow and Disallow rules
- Optional Crawl-delay and one or more Sitemap URLs
- Live preview that builds a valid robots.txt as you type
- One-click copy and download — save straight to your site root
- 100% in-browser — your rules never leave your device
Frequently asked questions
What is a robots.txt file?
A robots.txt file is a plain text file in your site root that tells search engine crawlers which URLs they may or may not request. It uses simple User-agent, Disallow and Allow directives, plus optional Crawl-delay and Sitemap lines.
Where do I put the robots.txt file?
It must sit in the root of your domain, reachable at https://example.com/robots.txt. A robots.txt in a subfolder is ignored. Download the file generated here and upload it to your web root.
Does robots.txt stop a page from being indexed?
No. Disallow blocks crawling, but a blocked URL can still appear in search results if other pages link to it. To keep a page out of the index, allow crawling and use a noindex meta tag or X-Robots-Tag header instead.
How do I block all bots from my whole site?
Use one group with User-agent: * and Disallow: / — the "Block all" preset does exactly this. To allow everything instead, leave Disallow empty, which the "Allow all" preset generates.
Should I add my sitemap to robots.txt?
Yes. Adding a Sitemap: line with the full URL helps crawlers discover your sitemap. It applies to all crawlers regardless of the user-agent groups, so add every sitemap you publish.