Robots.txt Tester

Search crawlers follow links and may revisit many variations of the same content, such as product filters or internal search pages. A robots.txt file gives these automated clients instructions about which URL paths to crawl. It can discourage requests to repetitive areas while leaving useful pages and their supporting resources available to participating crawlers.

The policy belongs at the site's root as /robots.txt. Its scope is an origin, meaning the combination of scheme, hostname and port. An HTTPS site, its HTTP counterpart and a separate subdomain can therefore have different policies. A file placed inside a subdirectory does not replace the root policy for that directory.

Rules are organized into groups. Each group names one or more crawlers through User-agent records, followed by Allow and Disallow rules. The crawler's identifying name is its product token, such as ExampleBot. A group addressed to * supplies fallback instructions for crawlers without a matching named group.

Path rules can cover more than one page. For example, a restriction on /catalog/ also covers paths below that prefix, while a more specific Allow rule can create an exception. Capitalization, trailing slashes, query strings and encoded characters can change which rule matches. The decision comes from the applicable groups and matching patterns, rather than simply the first rule in the file.

• Crawling concerns retrieving a URL. Permission under robots.txt does not guarantee that a crawler will visit it.
• Indexing concerns inclusion in search results. A blocked URL may still be discovered through links and appear in results without its content being crawled.
• Access control concerns who can retrieve protected content. Robots.txt provides no authentication or authorization, and its public list of paths can reveal locations that were otherwise hard to discover.

A crawl restriction can also prevent a search engine from reading a page's noindex directive. Google needs access to the page to see that instruction in its metadata or response header. Private content needs appropriate access controls, while search visibility needs the relevant indexing instructions.

A policy check describes the instructions for a particular crawler and URL. A crawler may have an older policy cached, encounter a retrieval error or apply additional behavior of its own. Those differences matter when investigating why an actual visit or search result does not match the current policy text.

How to Use This Tool:

Choose whether to inspect a public site's current policy or test complete policy text before publishing it. The initial setup contains example values that you can replace.

1. Choose Policy source. Use Fetch live robots.txt for retrieval, or Test pasted policy locally and replace the sample in Pasted robots.txt with the complete text you want to test.
2. Enter the site's HTTP or HTTPS origin in Public origin or Origin for pasted policy. Supply the scheme and host, with a port if needed, without a page path, query, fragment or credentials.
3. Enter one Crawler product token. The default is Googlebot. A value such as Googlebot/1.0 must be changed to Googlebot, because the field accepts a product token rather than a complete identification string.
4. Enter Paths to test, one per line, with between 1 and 20 entries. Use root-relative paths such as /catalog/?sort=price or full URLs on the selected origin. Include the query you want checked and remove any fragment.
5. Select Fetch robots.txt in live mode. No policy request is sent until you select it. Pasted mode updates the decisions locally as you edit, without a fetch action.
   If the summary says Check inputs, correct the first displayed warning. If a live request fails or is cancelled, retry before relying on a report from that request.
6. Review Path decisions and its reasons. For an unexpected rule decision, compare the indicated group and rule lines with Policy source. Changing the origin or policy source clears a fetched result; changing only the crawler or paths rechecks the policy already available.

Interpreting Results:

Read each Decision together with its reason. Allowed can come from an Allow rule, the absence of a matching restriction or the stated HTTP 4xx assumption. Disallowed can come from a rule or an unreachable-policy assumption. Indeterminate means the retrieval did not provide a complete usable policy for a decision.

For a rule-based result, Group / rule line identifies the group's first User-agent line and the deciding rule's line in the policy. A missing Deciding rule is expected for default permissions and retrieval-based assumptions. Confirm an unexpected result against the actual policy text, especially when a successful response contains few usable rules.

Rule competition compares the winning pattern with the strongest matching pattern that would give the opposite decision. Equal lengths can produce an Allow tie. A zero opposite value means there is no matching opposite rule; the chart is not a confidence score or a measure of how likely a crawler is to visit.

Ignored records distinguishes malformed rules from unsupported records such as Sitemap and Crawl-delay. Unsupported records do not determine these path decisions, but may have meaning to other crawlers or systems. The total includes every ignored record, while the display keeps the first 200 records and the first 512 characters of each.

Policy source initially previews 30 lines. Under Advanced, Source preview offers 10, 30 or 100 lines, with each displayed line limited to 2,048 characters. Matching and policy text copies or downloads use the complete accepted policy, including text outside that preview.

Selecting a named crawler chooses its policy groups using the same rules for every token. It does not reproduce that crawler's cache, retries or indexing decisions. The tested pages are not fetched, so an allowed result does not confirm that a page exists or responds successfully. Check the page response and actual crawler records separately when diagnosing live crawling.

Technical Details:

The Robots Exclusion Protocol, specified in RFC 9309, separates crawler-group selection from path matching. A very specific rule has no effect if it belongs only to a group that does not apply to the selected crawler.

Rule Core

Once a complete pasted or fetched policy is available, the decision follows this order.

1. Select every group containing an exact, case-insensitive match for the product token and combine its rules. If there are no exact matches, combine the groups addressed to *. Separate wildcard groups are not added when an exact match exists, even if the matching named group has no usable rules.
2. Permit the bare path /robots.txt implicitly. A query-bearing path such as /robots.txt?version=1 proceeds through ordinary rule matching.
3. Compare each selected rule with the normalized path and query, beginning at the first path character. Path comparisons are case-sensitive, so /Admin and /admin remain different.
4. Choose the matching rule with the greatest normalized pattern length. If Allow and Disallow tie at that length, Allow wins. For tied rules with the same directive, the earlier source line supplies the displayed evidence.
5. Allow the path if no selected rule matches. This also covers an empty complete policy, a matching group without usable rules, or the absence of both an exact-token group and a wildcard group.

Directive names are case-insensitive. Consecutive User-agent records can share the following rules, and another User-agent record after an Allow or Disallow record begins a new group. Blank lines, comments and unsupported records do not split an existing group. A leading Unicode byte-order mark is ignored.

Each # begins a comment extending to the end of its line. Empty Allow or Disallow values impose no restriction and are reported as ignored. Rules before a User-agent group, malformed records and invalid patterns are also ignored while the remaining usable rules are retained. Recognized path patterns start with / or * and cannot contain raw spaces, control characters or malformed percent escapes.

Prefixes, wildcards and query strings

A plain pattern matches a prefix, not a directory boundary. A wildcard broadens the match, while a final dollar sign requires the match to reach the end of the path and query.

These patterns are not regular expressions. A $ at the end of a pattern acts as the end marker. Use %24 when a dollar sign should match as text.

Unicode, percent encoding and pattern length

Equivalent spellings are compared using normalized URL octets. An octet is a byte. Percent-encoded unreserved ASCII letters, digits and -._~ are decoded for comparison. Non-ASCII text is represented through its UTF-8 percent-encoded bytes, while escaped reserved separators retain their distinction from literal separators.

Pattern octets measures the normalized percent-encoded comparison pattern. An ordinary ASCII character contributes one, and each retained percent escape contributes three. Wildcard stars and a terminal dollar sign also each contribute one. Both /café and /caf%C3%A9 therefore have a count of 10, while /*.gif$ has a count of 7. Use this count for rule precedence; visible character counts and raw UTF-8 file sizes measure different things.

The displayed Path contains the path and query after URL parsing. Dot segments can be resolved during that parsing, so the displayed path is the value used for the comparison. Fragments are rejected rather than silently removed.

Live retrieval and fetch outcomes

Live retrieval starts at the selected origin's root robots.txt and follows at most five redirects. A policy reached through permitted redirects is applied to the original origin, even if the final policy address differs. Effective policy URL shows the address reached when one is available.

The retrieval outcome is checked before path rules. In the following table, HTTP statuses belong to the robots.txt request, not to any tested page.

These retrieval decisions apply before the implicit permission for the bare robots.txt path.

An input error, cancellation or failure to obtain a valid response from the checking service can leave no completed path report at all. The absence of a report supplies no permission decision. There is no fallback to an older successful policy after a new fetch fails.

Input boundaries and complete-policy limits

Validation keeps each batch tied to one origin and one crawler. Limits apply to the entered text before matching.

All accepted policy text participates, including rules beyond 500 KiB. An oversized policy is rejected as a whole rather than tested from a shortened prefix. The 2 MiB limit belongs to this checker and does not establish how much text a particular crawler will process.

Privacy Notes:

Rule matching takes place in the browser in both modes. Live mode additionally retrieves the public policy when requested.

• The live fetch sends only the selected origin as its submitted input to the retrieval service. The crawler token, test paths and pasted policy are not included in that request. Live retrieval is for public destinations and does not provide authenticated access to private sites.
• Pasted policy text is processed in the current tab and is not submitted by the local test. Shared setup links exclude pasted policy text, fetched policy text and fetched results.
• The shareable setup includes the origin, policy mode, crawler token, test paths and preview length. Paths and query values can therefore be disclosed through a shared page URL. Remove sensitive values before sharing it.

Worked Examples:

References:

• RFC 9309: Robots Exclusion Protocol, RFC Editor, September 2022.
• Introduction to robots.txt, Google Search Central.
• Block Search indexing with noindex, Google Search Central, December 10, 2025.