CodeNimbleToolsFree Online Developer Tools
Explore tools

Robots.txt Generator

Create a clean robots.txt file with crawl rules and sitemap location.

Tool input is processed locally in your browser and is not submitted to CodeNimbleTools. Avoid pasting production secrets into any device or website you do not fully trust.


  





  

    
About this tool
A robots.txt file gives compliant crawlers instructions about which URL paths they may request. The Robots.txt Generator creates a user-agent block with Allow and Disallow rules, an optional crawl delay and a sitemap declaration, then lets you download the result as a plain-text file.
Robots.txt is not an access-control system: blocked URLs may still be discovered, and private data must be protected by authentication and server permissions. Do not block CSS or JavaScript that search engines need to render public pages, and remember that different crawlers do not all support every directive. For page-level index/follow decisions, use a robots meta tag or HTTP header; the Meta Tag Generator can create that markup.
    
How to use it
  1. Enter * to target all compliant crawlers or name a specific user agent.
  2. Add one Disallow path per line for areas that should not be crawled.
  3. Add narrower Allow paths only when an allowed subsection sits inside a broader blocked path.
  4. Enter the absolute sitemap URL and use crawl delay only when the target crawler supports it.
  5. Download robots.txt, upload it to the domain root, and test important URLs with search-console tools.
    
Useful ways to apply it
  • Blocking crawler access to internal search results, staging-like folders or duplicate utility paths.
  • Advertising the location of an XML sitemap.
  • Creating separate rule groups for a specific crawler.
  • Reviewing an intended policy before replacing a live robots file.
  • Teaching why crawl control is different from authentication and noindex.

          
Worked example

      

        
Example input
Disallow /admin/; Sitemap https://example.com/sitemap.xml

        
Expected output
User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml

      

    
          
Edge cases to check

      
  • robots.txt controls crawling, not access control.
  • Unsupported directives may be ignored.
  • Blocking a URL does not guarantee deindexing if the URL is discovered elsewhere.

    
    
Known limitations

    
Never use robots.txt to protect secrets. Validate generated rules for your actual crawler goals.


    

      
How this tool was reviewed

      
Reviewed against the Robots Exclusion Protocol and common crawl-control patterns.

      
Last reviewed: September 3, 2026 · Maintainer: William

              
Standards and technical references

        
            
See the site-wide testing methodology for fixture categories, privacy checks and correction policy.

    

  

    

    
Frequently asked questions

          
Where must robots.txt be uploaded?
It belongs at the root of the exact host, such as https://example.com/robots.txt. A file in a subfolder does not control the whole host.

          
Does Disallow remove a page from search results?
Not reliably. It prevents crawling by compliant bots, but a URL can still be indexed from links. Use noindex while the page remains crawlable when removal from results is the goal.

          
Can robots.txt protect private files?
No. Anyone can read the file, and malicious bots may ignore it. Protect private content with authentication and server rules.

          
Should I include a sitemap line?
It is useful and accepted by major search engines. Use the full absolute URL of the XML sitemap.

      

    
 Privacy
Tool input is processed locally in your browser and is not submitted to CodeNimbleTools. Avoid pasting production secrets into any device or website you do not fully trust.