Robots.txt Generator
Build a valid robots.txt with per-agent allow/disallow rules.
Files and text are processed locally in your browser. Tools that must reach the network say so explicitly.
About the Robots.txt Generator
Create a correct robots.txt without memorising the syntax. The Toolcenter Robots.txt Generator builds per-user-agent rule groups with allow and disallow paths, optional crawl-delay, sitemap references and a preferred host line. Presets cover “allow everything”, “block everything”, common e-commerce exclusions and blocking AI crawlers.
How to use it
- 1Start from a preset or an empty rule group.
- 2List user-agents and their disallow/allow paths.
- 3Add your sitemap URL(s).
- 4Copy or download robots.txt to your site root.
Key features
- Multiple per-agent rule groups
- Allow, disallow and crawl-delay directives
- Sitemap and preferred-host lines
- Presets including block-AI-crawlers
- Live valid output
Frequently asked questions
Where does robots.txt go?+
At the root of your domain, reachable at https://yourdomain.com/robots.txt. It only applies to that host and scheme.
Can I block ChatGPT and other AI bots?+
Yes. The “block AI crawlers” preset disallows GPTBot, CCBot, Google-Extended, anthropic-ai and ClaudeBot.
Does Disallow guarantee a page stays out of Google?+
No. It stops crawling, but a blocked URL can still be indexed if linked elsewhere. Use a noindex meta tag to keep it out.
Is anything uploaded?+
No. The file is generated in your browser.
Also searched as
Related seo & web
Meta Tag Generator
Generate title, description, Open Graph and Twitter card tags.
SERP Snippet Preview
Preview how your page looks in Google on desktop and mobile.
Keyword Density Checker
Analyze term frequency, n-grams and stop-word ratio.
Keyword Research Tool
Expand a seed keyword into questions, modifiers and long-tails.
Sitemap Generator
Crawl a site or paste URLs to build an XML sitemap.
Website Speed Test
Measure load time, page weight and request count for any URL.