How to use it
- Choose the Default for all crawlers and type the disallowed paths, one per line.
- Add allowed paths for folders that must stay open inside a blocked area, and enter your sitemap addresses.
- Choose what to do with the AI crawlers: add no group, block all of them, or allow all of them.
- Copy the file or download it, and save it as robots.txt in the top folder of your site.
How a crawler reads the file
A crawler obeys the group whose user-agent line names it. If no group names it, the crawler obeys the group that starts with an asterisk, which applies to all crawlers (RFC 9309, section 2.2.1). Google's robots.txt page says that the group with the most specific user agent that matches the crawler is used, and that other groups are ignored.
When a disallowed path and an allowed path both match an address, the longer path wins. If they are equally long, the allowed path wins (RFC 9309, section 2.2.2). Google says it uses the least restrictive rule in that case.
- Google requires the file in the top-level directory of a site, and its name must be robots.txt.
- Google ignores lines it does not understand and reads at most 500 kibibytes of the file.
The AI crawlers in this tool
Each token below is the name that its operator gives in its own documentation, except Bytespider, for which no official page was found. The purposes are as the operators describe them.
| Token | Operator and purpose | Operator's page |
|---|---|---|
| GPTBot | OpenAI: crawls content that may be used in training its generative AI foundation models | OpenAI bots page |
| ClaudeBot | Anthropic: collects web content that could contribute to training | Anthropic crawlers help page |
| Google-Extended | Google: controls whether crawled content may feed AI model development; it does not affect Search inclusion or ranking | Google crawler documentation |
| PerplexityBot | Perplexity: surfaces and links websites in its search results, and is not used to crawl content for AI foundation models | Perplexity bots guide |
| CCBot | Common Crawl: builds an open repository of web crawl data; robots.txt is the way to opt out | Common Crawl CCBot page |
| Applebot-Extended | Apple: controls whether content may be used to train Apple's foundation models | Apple's About Applebot page |
| Bytespider | ByteDance: not confirmed, because no official crawler page was found | None found |
| meta-externalagent | Meta: training foundation AI models or indexing content directly | Meta web crawlers page |
What robots.txt does not do
- It does not keep a page out of Google. Google's introduction says that robots.txt is not a mechanism for keeping a web page out of Google; use a noindex rule or password protection for that.
- It does not stop a crawler that chooses to ignore it. It only asks crawlers to keep away from the paths named in it.
- Google ignores Crawl-delay, so the crawl delay that you set here has no effect on Google.
Worked example: a shop that blocks AI crawlers
The default inputs allow all crawlers, close /admin/, /checkout/ and every address with a sort parameter, and keep /admin/help/ open, because that allowed path is longer than the disallowed /admin/. The AI crawlers are blocked, and the two sitemaps are listed at the end.
The table shows which line of the file decides for three addresses.
| Crawler | Address | Rule that decides |
|---|---|---|
| Googlebot, which has no group of its own | /admin/login | Disallow: /admin/ |
| Googlebot, which has no group of its own | /admin/help/faq.html | Allow: /admin/help/ |
| GPTBot | /products/shirt.html | Disallow: / |
Frequently asked questions
Does robots.txt keep a page out of Google search results?
No. Google's introduction to robots.txt says the file is not a mechanism for keeping a web page out of Google. To keep a page out, use a noindex rule or password protection.
Why does each AI group repeat the disallowed paths?
A crawler reads only the group that names it, and the group for all crawlers is ignored for it. The allow setting therefore repeats your disallowed paths under its own Allow: / line, so those paths stay closed for the AI crawler.
Does Google follow the Crawl-delay line?
No. Google's robots.txt page says that other fields such as crawl-delay are not supported. Anthropic says its crawlers support the non-standard Crawl-delay extension.
Does blocking Google-Extended affect my rankings in Google Search?
No. Google's crawler documentation says that Google-Extended does not impact a site's inclusion in Google Search, and that it is not used as a ranking signal.
Is the Bytespider token confirmed by its operator?
No. When this tool was checked, no ByteDance crawler page was found, so the token comes from third-party sites. Look for requests with that name in your server logs before you rely on the rule.
Where should I put the file?
In the top-level folder of the site, so that it opens at the root address under the name robots.txt. Google's page says that the file must be placed in the top-level directory of a site.
Does the tool store what I type or send it to a server?
No. The file is written in your browser. Nothing that you type is sent to us or stored.
Sources
- Google Search Central: how Google interprets the robots.txt specification
- Google Search Central: introduction to robots.txt
- Google Search Central: Google's crawlers, including Google-Extended
- RFC 9309: Robots Exclusion Protocol
- OpenAI: bots and crawlers, including GPTBot
- Anthropic: crawlers, including ClaudeBot
- Perplexity: bots, including PerplexityBot
- Common Crawl: CCBot
- Apple: About Applebot, including Applebot-Extended
- Meta: web crawlers, including meta-externalagent
Figures and text last checked on Tuesday, November 10, 2026.