Robots.txt Generator and Tester
Build a robots.txt with user-agent groups, Allow and Disallow rules and sitemaps, test which rule matches a URL, or parse an existing file. Runs in your browser.
How to use Robots.txt Generator
- 1Pick a preset or add user-agent groups and Allow or Disallow rows, then copy or download the robots.txt.
- 2Switch to Test, paste a robots.txt, and enter a user agent and a URL path to see whether it is allowed and which rule matched.
- 3Use Parse on a file already live on your site to catch rules outside a group, unknown directives and a byte-order mark.
What makes it better
- Presets: allow all, block all for staging, block AI training crawlers, WordPress, Next.js and e-commerce
- Test follows Google's rules: the longest matching path wins, Allow wins a tie, * is a wildcard and a final $ anchors the end
- Warns about paths that do not start with / or *, files over the 500 KiB that Google reads, and secret-looking paths
- Parse shows groups, rules and sitemaps with line numbers, plus warnings for typos and unsupported directives
- Build, Test and Parse share one input, so the file you tested is the file you parse
About Robots.txt Generator
Robots.txt Generator builds the file from user-agent groups, Allow and Disallow rows, crawl-delay and sitemap lines, with presets for WordPress, Next.js, shops, staging sites and AI crawlers. The Test mode answers whether a crawler may fetch a path and names the rule that decided it, using Google's longest-match rule. The Parse mode reads an existing file and lists problems by line number.
Common uses
- Block a staging site from search engines before launch, and swap in the production file afterwards.
- Ask GPTBot, CCBot and other AI crawlers to stay away from your content while Googlebot keeps crawling.
- Check why a product URL with a filter parameter is not being crawled by trying the exact URL against your rules.
- Review a robots.txt inherited from an agency to find leftover rules that block CSS, JavaScript or whole sections.
Example
Input
# my shop user-agent: * disallow: /admin/ disallow: /search allow: /admin/help sitemap: https://example.com/sitemap.xml USER-AGENT: GPTBot Disallow: /
Result
User-agent: * Disallow: /admin/ Disallow: /search Allow: /admin/help User-agent: GPTBot Disallow: / Sitemap: https://example.com/sitemap.xml
Good to know
- robots.txt is a request, not access control. Well-behaved crawlers follow it, bad ones ignore it, and the file itself is public, so never list private URLs in it. Protect them with a login.
- Disallow stops crawling, not indexing: a blocked URL can still show up in search results if other sites link to it. Use a noindex meta tag or header on a page that crawlers are allowed to fetch.
- Google ignores Crawl-delay, and the AI crawler list changes often. The tester matches the way Google documents it; other crawlers can differ in small details such as how they pick between groups.
Robots.txt Generator FAQ
Where do I put the robots.txt file?
Upload it to the root of the host it applies to, so it is reachable at https://example.com/robots.txt. A file in a subfolder is ignored, and each subdomain such as blog.example.com needs its own robots.txt.
How do I block AI crawlers like GPTBot in robots.txt?
Choose the Block AI training crawlers preset. It writes one group naming GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, CCBot, Google-Extended, PerplexityBot, Bytespider, Amazonbot and Applebot-Extended with Disallow: /. Compliance is voluntary and new crawler names appear regularly.
What do * and $ mean in a Disallow rule?
An asterisk matches any run of characters, so /*.pdf blocks every URL that contains .pdf. A dollar sign at the very end means the URL must end there, so /*.pdf$ blocks PDF files but not /file.pdf?download=1.
Which rule wins when Allow and Disallow both match?
The rule with the longer path is more specific and wins. If both are the same length, Allow wins. For example, Allow: /shop/sale beats Disallow: /shop/ for /shop/sale/shoes, and the Test mode shows the winning line.
Does robots.txt stop a page from appearing in Google?
No. It only asks crawlers not to fetch the page. Google can still list the bare URL if other pages link to it. To keep a page out of results, let Google crawl it and add a noindex robots meta tag or an X-Robots-Tag header.