robots.txt Generator
Generate a robots.txt file for your site using common directives such as User-agent, Allow, Disallow, Crawl-delay, and Sitemap.
robots.txt Generator tool
Use one path per line. Paths should usually begin with /.
| Metric | Value |
|---|
Privacy: generation runs locally in your browser. No rules are stored or transmitted.
Enter a user-agent and optional allow/disallow rules, then generate a robots.txt file that can be placed at your site root.
- User-agent: which crawler the rules apply to
- Allow / Disallow: crawl path rules
- Sitemap: helps crawlers discover your sitemap
Examples
- Public site: allow all, include sitemap
- Block admin: disallow
/admin/ - Selective crawl: combine allow and disallow directives
FAQ
- What does this robots.txt generator create?
It generates robots.txt directives such as User-agent, Allow, Disallow, Crawl-delay, and Sitemap.
- Can I include multiple allow or disallow rules?
Yes. Enter one path per line and the generator will output each directive separately.
- Does this validate sitemap URLs?
Yes. If a sitemap URL is provided, the tool checks for a valid http or https URL format.
- Does robots.txt guarantee blocking from search results?
No. robots.txt is mainly for crawl control, not guaranteed de-indexing.
- Are my rules stored?
No. Everything runs locally in your browser.