Robots.txt Generator

About this tool

A robots.txt file gives compliant crawlers instructions about which URL paths they may request. The Robots.txt Generator creates a user-agent block with Allow and Disallow rules, an optional crawl delay and a sitemap declaration, then lets you download the result as a plain-text file.

Robots.txt is not an access-control system: blocked URLs may still be discovered, and private data must be protected by authentication and server permissions. Do not block CSS or JavaScript that search engines need to render public pages, and remember that different crawlers do not all support every directive. For page-level index/follow decisions, use a robots meta tag or HTTP header; the Meta Tag Generator can create that markup.

How to use it

1. Enter * to target all compliant crawlers or name a specific user agent.
2. Add one Disallow path per line for areas that should not be crawled.
3. Add narrower Allow paths only when an allowed subsection sits inside a broader blocked path.
4. Enter the absolute sitemap URL and use crawl delay only when the target crawler supports it.
5. Download robots.txt, upload it to the domain root, and test important URLs with search-console tools.

Useful ways to apply it

• Blocking crawler access to internal search results, staging-like folders or duplicate utility paths.
• Advertising the location of an XML sitemap.
• Creating separate rule groups for a specific crawler.
• Reviewing an intended policy before replacing a live robots file.
• Teaching why crawl control is different from authentication and noindex.