robots.txt Guide: How to Control Search Engine Crawlers

The robots.txt file is a simple but powerful tool that tells search engine crawlers which parts of your website they should or shouldn't access. Understanding robots.txt is essential for SEO and controlling how your site appears in search results.

What is robots.txt?

robots.txt is a plain text file located at your website's root (example.com/robots.txt) that follows the Robots Exclusion Protocol. It provides instructions to web crawlers about which URLs they can access.

robots.txt Syntax

User-agent

Specifies which crawler the rules apply to:

User-agent: *           # All crawlers
User-agent: Googlebot   # Google only
User-agent: Bingbot     # Bing only

Disallow

Tells crawlers NOT to access certain paths:

Disallow: /private/
Disallow: /admin/
Disallow: /tmp/

Allow

Explicitly allows access (overrides Disallow):

Disallow: /private/
Allow: /private/public-page.html

Sitemap

Points crawlers to your XML sitemap:

Sitemap: https://example.com/sitemap.xml

Common robots.txt Examples

Block Everything

User-agent: *
Disallow: /

Allow Everything (or omit robots.txt)

User-agent: *
Disallow:

Block Specific Directory

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /private/

Block Specific Crawler

User-agent: BadBot
Disallow: /

User-agent: *
Disallow:

Important Considerations

robots.txt is Public

Anyone can view your robots.txt file. Don't list sensitive directories – you're advertising their existence!

Not a Security Measure

robots.txt is a suggestion, not a command. Malicious bots will ignore it. For true access control, use authentication or firewall rules.

Blocking != De-indexing

If a page is already indexed, blocking it in robots.txt won't remove it from search results. Use the noindex meta tag instead.

Common Mistakes

• Blocking CSS/JS - Can hurt SEO; Google needs to render pages
• Blocking sitemaps - Self-defeating
• Typos in paths - Paths are case-sensitive
• Missing trailing slashes - /admin is different from /admin/
• Blocking entire site accidentally - Be careful with Disallow: /

Using Our robots.txt Checker

Enter any domain to instantly analyze its robots.txt file:

• View all user-agent rules
• See allowed and disallowed paths
• Find sitemap references
• Identify potential issues

Use this tool to audit your own site or analyze competitors' crawling strategies!