ApneSoftware← All Tools

🔎 Robots.txt Analyzer

Paste your robots.txt file to check for errors, understand all directives, and test whether specific URLs are allowed or blocked for any bot.

⌨️ Ctrl+Enter to analyze

🔎 How to Analyze Your Robots.txt

This tool parses your robots.txt file and shows you every rule, grouped by User-agent. Use the URL tester to check if any specific URL would be crawled or blocked for a given bot.

Common Robots.txt Mistakes

  1. Accidentally blocking all bots with Disallow: /
  2. Blocking CSS/JS files (prevents Google from rendering pages)
  3. Case-sensitive paths (robots.txt paths are case-sensitive)
  4. Missing trailing slash on directory rules
  5. Sitemap URL pointing to wrong path

🔎 Hindi Guide / हिंदी गाइड

यह tool आपकी robots.txt file को analyze करता है और सभी rules को समझकर बताता है। URL Tester से आप check कर सकते हैं कि कोई specific URL allow है या block।

Common Mistakes

  1. Disallow: / — सभी pages block हो जाते हैं! बहुत common गलती।
  2. CSS/JS files block करना — Google page render नहीं कर पाता
  3. Sitemap URL wrong होना
  4. Case-sensitive paths में गलती