Robots.txt Generator
Generate a robots.txt file to control crawler access to specific areas of your website.
- Specify Allow and Disallow rules
- Add sitemap URL
- Download or copy the generated file
- Local processing only
The user-agent to which rules apply. Use * for all crawlers.
Disallow Rules
Paths that crawlers should not access. Use one path per field.
Allow Rules
Specific paths that crawlers are allowed to access. Use one path per field.
Full URL to your sitemap.xml file.
This generator runs locally in your browser. Your input is never sent to any server.
What is robots.txt?
The robots.txt file is a text file placed in your website's root directory that provides instructions to web crawlers about which areas of your site they can or cannot access.
How to use robots.txt
Place the robots.txt file at the root of your domain (e.g., https://example.com/robots.txt). The file applies to the entire domain and subdirectories unless specified otherwise.
Understanding directives
- User-Agent: Specifies which crawler the rules apply to (* means all crawlers)
- Disallow: Blocks crawlers from accessing specific paths or files
- Allow: Explicitly permits access to specific paths (often used with wildcards)
- Sitemap: Provides the location of your sitemap.xml file
Important limitations
Robots.txt is not a security or access control mechanism. It does not prevent unauthorized access to content and should not be used to protect sensitive files. Malicious bots may ignore robots.txt, and search engines may still index blocked pages if linked from other sources.
Best practices
- Use robots.txt to manage crawl budget, not for privacy or security
- Test your robots.txt file with search console tools
- Be specific with paths to avoid unintended blocking
- Remember that blocking crawling ≠ preventing indexing
FAQ
Does robots.txt guarantee that pages won't be indexed?
No. Search engines may still index a page even if blocked by robots.txt if other pages link to it. To prevent indexing, use the noindex meta tag or X-Robots-Tag header.
Can robots.txt hide private content?
No. Robots.txt is publicly accessible and should not be used to protect private or sensitive content. Use proper authentication and access controls instead.
How do I test my robots.txt file?
You can use search console tools provided by search engines to test how their crawlers interpret your robots.txt file.
Related SEO tools
Complement your technical SEO with comprehensive analysis
Beyond robots.txt, use Mangools for broader SEO research and analysis to optimize your site's visibility.