Robots.txt Generator
Create and customize your robots.txt file with our visual builder. Control which crawlers can access your site, block AI bots, set crawl delays, and add sitemap references.
Quick Presets
Sitemaps
Preview
User-agent: * Allow: /
Save this file as robots.txt in the root directory of your website (e.g., https://example.com/robots.txt)
About Robots.txt
The robots.txt file is a text file that tells web crawlers which pages or sections of your site they can or cannot access. It's placed in the root directory of your website and follows the Robots Exclusion Protocol standard.
How It Works
When a crawler visits your site, it first checks for a robots.txt file at yourdomain.com/robots.txt. The file contains directives that specify which parts of your site can be crawled. Well-behaved bots will respect these rules, though malicious bots may ignore them.
Key Directives
User-agent: Specifies which crawler the rules apply to (use * for all). Disallow: Blocks access to specified paths. Allow: Explicitly permits access (useful with wildcards). Crawl-delay: Requests a delay between requests. Sitemap: Points to your XML sitemap.
Blocking AI Crawlers
With the rise of AI systems that scrape content for training, many site owners now block AI crawlers. Common AI user agents include GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google AI), CCBot (Common Crawl), and others. Our generator includes a preset to block all known AI crawlers.
Best Practices
Don't use robots.txt to hide sensitive content—it's publicly visible. Use it to prevent crawling of duplicate content, admin pages, or resource-heavy sections. Always test your robots.txt before deploying. Include a sitemap reference for better SEO.
Common Directives:
- User-agent: * - All crawlers
- Disallow: / - Block everything
- Allow: / - Allow everything
- Disallow: /admin/ - Block folder
- Crawl-delay: 10 - 10s between requests
- Sitemap: - XML sitemap URL
- Disallow: *.pdf$ - Block PDF files
- # Comment - Add notes
Important: robots.txt is a request, not a command. It works on the honor system—reputable search engines and bots respect it, but malicious bots can ignore it. Never rely on robots.txt for security.
Need Help With SEO & Crawl Optimization?
Our team can help optimize your site's crawlability, fix indexing issues, and ensure search engines can properly discover your content. Book a free discovery call.