robots.txt Checker

Analyze robots.txt files to verify crawler directives, sitemap references, and identify potential SEO issues.

100% client-side No signup Free forever

About robots.txt

The robots.txt file tells search engine crawlers which pages or files they can or cannot request from your site.

  • User-agent - Specifies which crawler the rules apply to
  • Disallow - Paths the crawler should not access
  • Allow - Paths the crawler can access (overrides Disallow)
  • Sitemap - Location of XML sitemaps
// how to use

How to Use robots.txt Checker

  1. Enter the website domain (e.g., example.com)
  2. Click "Check robots.txt" to fetch the file
  3. View all rules for different user agents
  4. Check allowed and disallowed paths
  5. Identify sitemap references and crawl-delay settings
// about

About robots.txt Checker

The robots.txt file guides search engine crawlers through your website, indicating which areas to index and which to skip. Misconfiguration can inadvertently block important content or waste crawl budget on irrelevant pages. Our checker analyzes robots.txt files for proper syntax and potential issues.

What We Analyze

  • Syntax Validation: Proper formatting of User-agent, Disallow, Allow, and Sitemap directives
  • Crawler Rules: What's blocked and allowed for different bots (Googlebot, Bingbot, etc.)
  • Sitemap References: Whether sitemaps are properly declared and accessible
  • Crawl-delay Directives: Rate limiting instructions for crawlers
  • Potential Conflicts: Overlapping rules that might cause unexpected behavior

Common Mistakes We Detect

Accidentally blocking CSS or JavaScript files prevents search engines from rendering pages properly. Blocking entire directories like /images/ hides visual content from image search. Using "Disallow: /" blocks the entire site—sometimes left from development and forgotten. Malformed paths (missing leading slashes) may not work as intended.

SEO Impact

Google respects robots.txt but may still index blocked pages if external links point to them—the content just won't be crawled. For truly private content, authentication is required rather than robots.txt alone. Our tool helps ensure your robots.txt achieves its intended purpose without unintended consequences.

// faq

Frequently Asked Questions

What does the robots.txt checker analyze?
We validate syntax, test specific URLs against rules, identify blocked important pages, check sitemap references, and highlight potential issues. The tool explains what each directive means and its impact on crawling.
How do I test if a specific URL is blocked?
Enter the URL to test against the robots.txt rules. We show whether it's allowed or blocked, and which rule matches. Test multiple URLs to verify your configuration works as intended for different page types.
What are common robots.txt mistakes?
Common errors: blocking CSS/JS files (breaks rendering), blocking important content accidentally, using wrong paths (case-sensitive!), incorrect wildcard usage, and forgetting that blocked pages can still be indexed if linked elsewhere.
How do I block search engines but allow specific bots?
Use specific User-agent directives: "User-agent: Googlebot" with "Allow: /" for Google, then "User-agent: *" with "Disallow: /" for others. Order matters - specific rules override general. We help construct correct multi-bot configurations.
Why isn't robots.txt blocking pages from Google?
robots.txt prevents crawling, not indexing. If other sites link to blocked pages, Google may index URLs (showing "No information available"). For true blocking, use noindex meta tags or require authentication. robots.txt is for crawl management only.
How do I verify my sitemap is referenced correctly?
We check for Sitemap: directives and validate the referenced URLs. The sitemap should be accessible (200 status), valid XML, and contain your important URLs. Missing sitemap references mean search engines might not find all your content.