Understanding robots.txt: Control Search Engine Crawling

A tiny text file in your website's root directory has enormous power over how search engines interact with your site. Understanding robots.txt helps you control what gets indexed—and what doesn't.

What Is robots.txt?

Robots.txt is a file that tells web crawlers which pages they can and cannot access. It follows the Robots Exclusion Protocol, a standard since 1994.

Location: Always at your domain root

• Correct: https://example.com/robots.txt
• Incorrect: https://example.com/pages/robots.txt

Create yours with our robots.txt generator.

How Search Engines Use It

1. Crawler visits your site
2. First checks for robots.txt at the root
3. Reads rules applicable to it
4. Respects or ignores based on the crawl directive

Important: robots.txt is a suggestion, not a security measure. Well-behaved crawlers follow it; malicious ones ignore it completely.

Basic Syntax

User-agent

Specifies which crawler the rules apply to:

# Apply to all crawlers
User-agent: *

# Apply to Googlebot only
User-agent: Googlebot

# Apply to Bing only
User-agent: Bingbot

Disallow

Blocks access to paths:

# Block entire site
Disallow: /

# Block specific directory
Disallow: /admin/

# Block specific file
Disallow: /private.html

Allow

Permits access (used to override Disallow):

# Block /images/ but allow /images/public/
User-agent: *
Disallow: /images/
Allow: /images/public/

Sitemap

Points crawlers to your sitemap:

Sitemap: https://example.com/sitemap.xml

Common Use Cases

Block Admin Areas

User-agent: *
Disallow: /admin/
Disallow: /wp-admin/
Disallow: /login/

Block Development Environments

User-agent: *
Disallow: /staging/
Disallow: /dev/
Disallow: /test/

Block Search Results Pages

User-agent: *
Disallow: /search
Disallow: /*?s=
Disallow: /*?q=

Block Parameter-Heavy URLs

User-agent: *
Disallow: /*?*
Allow: /

Rate Limit Crawling

User-agent: *
Crawl-delay: 10

(Note: Googlebot ignores Crawl-delay; use Search Console instead)

What NOT to Block

Avoid blocking:

• CSS and JavaScript: Google needs these to render pages
• Important images: Blocks image search traffic
• Sitemaps: You want crawlers to find these
• Canonical pages: Only block duplicates, not originals

robots.txt vs. Meta Robots

For pages you want crawled but not indexed, use:

<meta name="robots" content="noindex">

Testing Your robots.txt

Before deploying:

1. Use Google Search Console's robots.txt Tester
2. Check that important pages aren't blocked
3. Verify that intended blocks work
4. Test with different user-agents

Common Mistakes

Blocking Your Entire Site

# Accidentally blocks everything
User-agent: *
Disallow: /

One misplaced character can deindex your site.

Incorrect Path Matching

# Blocks /products/ but not /products
Disallow: /products/

# Blocks both
Disallow: /products

Blocking Important Resources

CSS/JS blocks cause rendering issues and may hurt rankings.

Using for Security

robots.txt is public—don't list sensitive URLs you want hidden. It actually advertises them!

Monitoring and Maintenance

• Review robots.txt after site structure changes
• Check Search Console for crawl errors
• Audit blocked resources periodically
• Update sitemap references when URLs change

Tools

• Robots.txt Generator - Create proper robots.txt files
• htaccess Generator - Server-level access control
• Meta Tag Generator - Page-level SEO control

A well-crafted robots.txt improves crawl efficiency and keeps private pages out of search results.