🤖 Robots.txt Generator

🤖 What is robots.txt?

The robots.txt file is a text file placed at the root of your website that tells search engine crawlers which pages they can and cannot access. It's the first file any crawler checks before visiting your site.

Important: robots.txt controls crawling, not indexing. A page blocked in robots.txt may still appear in search results if other pages link to it. To prevent indexing, use the noindex meta tag instead.

Key Directives

1. User-agent — Which bot the rules apply to (* = all)
2. Disallow — Paths the bot cannot crawl
3. Allow — Exceptions to disallow rules
4. Crawl-delay — Seconds to wait between requests
5. Sitemap — Location of your XML sitemap

Common Use Cases

1. Block /admin/ from all bots
2. Block /wp-admin/ and /wp-login.php
3. Block /search? query parameter pages
4. Block /cart/ and /checkout/ pages
5. Allow Googlebot but block others

🤖 Hindi Guide / हिंदी गाइड

robots.txt एक text file है जो आपकी website की root पर रखी जाती है। यह search engine crawlers को बताती है कि किन pages को crawl करना है और किन्हें नहीं।

Important Points

1. File का नाम exactly robots.txt होना चाहिए (lowercase)
2. यह file domain root पर होनी चाहिए: example.com/robots.txt
3. robots.txt crawling block करती है, indexing नहीं
4. Indexing रोकने के लिए noindex meta tag use करें
5. * (star) का मतलब सभी bots
6. Crawl-delay: Server load कम करने के लिए

Common Rules

1. Admin pages block करें: Disallow: /admin/
2. Search pages block करें: Disallow: /search?
3. Cart pages block करें: Disallow: /cart/
4. Sitemap add करें: Sitemap: https://example.com/sitemap.xml