Robots.txt Generator

Web crawlers can spend time on duplicate pages, search results, carts, account areas, generated filters, and other URLs that a site owner does not want routinely fetched. A root-level robots.txt file publishes crawl guidance for one protocol, host, and port. The HTTPS and HTTP versions of a host, or two different subdomains, therefore have separate policies.

The file groups rules beneath one or more crawler product tokens. A specific crawler group applies when its product token matches; the wildcard * group covers crawlers that have no more specific match. Inside the applicable group, the longest matching path wins. If equally specific Allow and Disallow rules match, Allow takes precedence under the Robots Exclusion Protocol.

Path matching begins at the first character of the URL path and is normally case-sensitive. A broad Disallow: /private/ rule can be narrowed with Allow: /private/help/. Reordering those two lines does not change which one wins because match length, not file order, controls the result.

• Allow permits a matching path inside the selected crawler group.
• Disallow asks a compliant crawler not to fetch a matching path.
• Sitemap declares an absolute sitemap location independently of crawler groups.
• Crawl-delay is a non-standard extension with crawler-specific support; Google does not support it.

Robots rules are neither authentication nor a reliable way to hide information. The file is public, non-compliant bots may ignore it, and blocked URLs can still be discovered through links or appear in search results. Private content needs access control, and indexing control may require an appropriate response header or page directive that the crawler can actually fetch.

A site-wide root block is useful for a staging origin only when that origin cannot be protected more strongly. Copying Disallow: / to production can remove the entire site from routine crawl access, so publication deserves the same review as a routing or firewall change.

How to Use This Tool:

Define the exact origin first, then write crawler groups around real URL paths rather than filesystem directories.

1. Choose a Policy preset or import an existing policy. Import accepts supported records only and replaces the current crawler groups after the whole pasted file passes structural checks.
2. Enter the Site origin as an HTTP or HTTPS origin without a path, query, fragment, or credentials, then add any absolute sitemap URLs.
3. Set each crawler group's product token and add Allow or Disallow paths beginning with /. Keep a wildcard group when unmatched crawlers should receive a fallback policy.
4. Enable Crawl-delay extension only for crawlers known to support it. Sorting rules changes their presentation but not longest-match behavior.
5. Read Publish checks, inspect the generated policy, and confirm that no unintended group contains an unqualified Disallow: / before publishing it as lowercase /robots.txt.

Interpreting Results:

Open crawl policy means no Disallow rule is present. Selective crawl policy means at least one path is disallowed. Root crawl block means the wildcard group contains Disallow: / without an equivalent Allow: /.

• A publish warning identifies a missing wildcard group, missing sitemap declaration, root block, or emitted Crawl-delay value. It does not predict how every crawler will behave.
• Rule counts describe the generated file. They do not prove that the live origin serves it with a successful plain-text response.
• Review the actual URL path and the longest matching rule when an exception appears ineffective; line order is not the deciding factor.

Technical Details:

The Robots Exclusion Protocol selects the crawler group with the most specific matching product token, then compares Allow and Disallow path patterns within that group. Path specificity is measured by matched octets, and a URL is allowed when no rule matches.

Rule Core:

Percent encoding and non-ASCII characters affect path comparison under RFC 9309. When URLs contain reserved or international characters, compare the encoded request path with the published rules instead of relying only on how the browser displays the URL.

Transformation Core:

Generation deduplicates non-empty sitemap and path lines, optionally sorts Allow and Disallow paths within each group, then emits comments, User-agent records, path rules, optional Crawl-delay records, and Sitemap records. Sorting affects readability only.

Import recognizes comments, User-agent, Allow, Disallow, Crawl-delay, and Sitemap records. Consecutive User-agent records share one rule block and are expanded into separate editable groups. An unsupported directive is not silently preserved.

Limitations and Privacy Notes:

Policy generation and import parsing happen in the browser, but the finished file is meant to be public. Comments and path rules can reveal administrative routes, customer names, unreleased features, or internal structure.

• Do not place secrets, credentials, or sensitive identifiers in comments, paths, or sitemap URLs.
• Use authentication and authorization for private content; crawler compliance is voluntary.
• Test the live response status, content type, encoding, and body after publication. This generator does not fetch or validate the deployed file.
• Crawler extensions differ. A valid emitted Crawl-delay record still has no effect on crawlers that ignore it.

Worked Examples:

References:

• RFC 9309: Robots Exclusion Protocol, RFC Editor, September 2022.
• How Google interprets the robots.txt specification, Google Crawling Infrastructure.
• Sitemaps XML format, sitemaps.org.
• How to create a robots.txt file for your website, Simplified Guide.