Robots.txt Generator
User-agent rules · Allow / Disallow · Crawl-delay · Sitemap · Free
Quick Presets
Agent Groups
Global Settings
robots.txt • output
Upload the generated robots.txt file to your website's root directory so it's accessible at https://yourdomain.com/robots.txt.
About the Robots.txt Generator
The robots.txt file lives at the root of your website and tells crawlers which pages they are allowed to index. Each block starts with User-agent (the bot) followed by Allow or Disallow rules. A * wildcard matches all bots. The Sitemap directive points crawlers to your XML sitemap.
User-agent: *
Applies rules to all crawlers. Use specific bot names (Googlebot, Bingbot) to target individual crawlers.
Disallow: /admin/
Prevents crawlers from indexing the /admin/ path and all its sub-pages.
Block AI Bots
GPTBot, Claude-Web, CCBot, and Google-Extended are AI training crawlers. Block them if you prefer not to have your content used for training.
Crawl-delay
Limits how fast a bot crawls your site. Useful if your server can't handle high crawl rates. Not honoured by Googlebot.