robots.txt Generator
Create robots.txt files to control search engine crawling. Allow or block specific pages and directories from indexing.
Generated robots.txt
Upload the robots.txt file to your website's root directory (e.g., https://example.com/robots.txt)
How to Use robots.txt Generator
How to Generate robots.txt
- Select crawlers: Choose which user-agents to configure (all, Googlebot, Bingbot, specific bots)
- Set Allow rules: Specify paths that should be crawled
- Set Disallow rules: Specify paths to block from crawling
- Add sitemap: Enter your XML sitemap URL(s)
- Set crawl delay: Optionally specify seconds between requests
- Generate: Create your robots.txt content
- Download/Copy: Save the file and upload to your site root
Where to Place robots.txt
Upload to your domain root: https://yoursite.com/robots.txt
It must be accessible at exactly this URL to work.
Features
- Visual rule builder interface
- Multiple user-agent configurations
- Allow and Disallow path settings
- Sitemap URL inclusion
- Crawl-delay settings
- Common preset templates
- Syntax validation
- Preview generated file
- Copy to clipboard
- Download as .txt file
- Test with major search engines
- Free with no registration
About robots.txt Generator
The robots.txt file is the first thing search engine crawlers check when visiting your site. It tells crawlers which pages to access and which to skip. A properly configured robots.txt protects private content and focuses crawler attention on important pages.
Robots.txt Directives
The file uses simple directives:
- User-agent: Which crawler the rules apply to (* means all)
- Disallow: Paths crawlers should not access
- Allow: Explicitly permit access (overrides Disallow)
- Sitemap: Location of your XML sitemap
- Crawl-delay: Seconds between requests (some crawlers)
Common Configuration Scenarios
Block admin areas: Disallow /admin/, /wp-admin/. Block search results: Disallow /search, /?s=. Block development: Disallow /staging/, /dev/. Allow everything: Just specify sitemap location.
Important Considerations
Robots.txt is a request, not enforcement—malicious bots ignore it. Don't rely on it for security. It's public, so blocking a path reveals that path exists. For sensitive content, use authentication instead. Googlebot respects robots.txt but may still index disallowed URLs if linked from other sites.
Testing Your Robots.txt
Use Google Search Console's robots.txt tester to verify your rules work as expected. Check that important pages aren't accidentally blocked and that private areas are properly excluded.