How to use it
- Paste your robots.txt file into robots.txt file.
- Enter the path or the full address in Path or address, and the name of the crawler in Crawler name.
- Read Allowed for this crawler, then the line in Rule that decides to see why.
How the decision is made
The tool first chooses the group. The user-agent lines that name the crawler, compared without regard to case, form one group. If none names it, the group for all crawlers (the asterisk) is used (RFC 9309, section 2.2.1).
Within that group, every rule whose pattern matches the address from its first character is compared. The longest pattern wins, and an allow rule wins a tie (RFC 9309, section 2.2.2). If no rule matches, the address is allowed. The file /robots.txt itself is always allowed.
- An asterisk (*) in a pattern matches any text, and a dollar sign ($) at the end of a pattern ends the match.
- Escapes of letters, digits and the characters - . _ ~ are decoded before the comparison, so %7E and ~ are the same.
- Comments start at a hash sign, and an empty Allow or Disallow value is ignored.
Where Google and the RFC agree
Google's robots.txt page describes the same rules: the group with the most specific matching user agent is used, the most specific rule based on the length of its path wins, and the least restrictive rule wins a conflict. The tool follows the RFC where the two texts differ in wording, and it finds no difference in these rules.
- Google matches the path and the query. The RFC does not say whether the query is part of the path; this tool matches it, as the example rows of Google's page do.
- The RFC does not say what an empty value means. Google ignores it, and so does this tool.
Worked example: three addresses on one file
The default file has a group for all crawlers and a group for Googlebot. Googlebot follows only its own group, so /drafts/ is blocked for it but not for other crawlers. The table shows the result for three requests.
Line 9 blocks /private/ for Googlebot. Line 5 is an allow rule that is longer than the disallowed /private/, so it wins for press-kit.html.
| Address | Crawler | Status | Rule that decides | Group that applies |
|---|---|---|---|---|
| /private/page.html | Googlebot | Blocked | Line 9: Disallow: /private/ | Line 7: User-agent: Googlebot |
| /private/press-kit.html | Bingbot | Allowed | Line 5: Allow: /private/press-kit.html | Line 2: User-agent: * |
| /drafts/new.html | Googlebot | Blocked | Line 8: Disallow: /drafts/ | Line 7: User-agent: Googlebot |
Frequently asked questions
Does this tool check the host name in a full address?
No. Only the path and the query are used. If the file has rules that name another host, this tool does not check whether they apply to the address you entered.
Why is the robots.txt file itself always allowed?
RFC 9309 says that the /robots.txt URI is implicitly allowed, so a crawler can always read the rules. The tool follows that rule and shows the result as the robots file.
What happens when two rules match the same address?
The rule with the longer pattern decides, because it is the more specific one. If the two patterns are equally long, the allow rule decides.
Why does a rule with an empty value have no effect?
The RFC grammar allows an empty value but does not say what it means. Google's page says that crawlers ignore a rule without a path, and this tool does the same.
Is my file sent to a server?
No. The file is read in your browser. Nothing that you paste is sent to us or stored.
Does a file longer than 500 KiB still work?
Only the first 500 KiB are read, as Google reads it. The tool says so when the file is longer, so you can see whether a rule at the end would be ignored.
Sources
- RFC 9309: Robots Exclusion Protocol
- Google Search Central: how Google interprets the robots.txt specification, with its path-matching examples
- Google Search Central: introduction to robots.txt
Figures and text last checked on Tuesday, November 10, 2026.