XML Sitemap Generator
★★★★★★★★★★5.0(0 ratings)👁 1❤ 0
Build a valid XML sitemap from a list of URLs. Set priority, change frequency, and last-modified date.
Robots.txt Generator
★★★★★★★★★★5.0(0 ratings)👁 1❤ 0
Generate a clean robots.txt with custom rules, AI-bot blockers, sitemap declaration, and popular presets.
Robots.txt Generator
User-agent: *
Allow: /
About This Tool
What is robots.txt?
The robots.txt file lives at yoursite.com/robots.txt and tells crawlers which parts of your site they may or may not access.
Common Use Cases
- Block crawlers from admin, login, or staging URLs
- Prevent indexing of duplicate pages
- Declare your XML sitemap URL
- Block AI training scrapers
Important
robots.txt is a polite request — malicious crawlers ignore it. Use server authentication or noindex meta tags to truly block content.
Frequently Asked Questions
Where do I upload robots.txt?
Place it at your domain root: yoursite.com/robots.txt. On WordPress, most SEO plugins (Yoast, Rank Math) let you edit it from the dashboard without FTP.
Will blocking AI scrapers actually work?
Major companies (OpenAI, Anthropic, Google) honor robots.txt. Less reputable scrapers may ignore it. For server-level enforcement, combine with IP blocking or Cloudflare WAF rules.
Should I block /wp-admin?
Yes, but allow /wp-admin/admin-ajax.php u2014 many WordPress features depend on it. Our default templates handle this correctly.