robots.txt Generator
Build a valid robots.txt from per-user-agent Allow/Disallow rules, a sitemap line and an optional crawl-delay, with presets including blocking AI crawlers.
User-agent: * Disallow: /admin/ Disallow: /cart Disallow: /*?* Sitemap: https://www.example.com/sitemap.xml
Place the file at https://yourdomain.com/robots.txt — it only applies to the host it is served from. Disallowwith an empty value means “allow everything”; Disallow: / blocks the whole site. robots.txt is a crawling directive, not an access control: it asks well-behaved crawlers not to fetch a path, but the pages can still be indexed if linked elsewhere — use noindex or authentication for that. Crawl-delay is ignored by Googlebot (set the rate in Search Console).
How robots.txt works — and what it cannot do
robots.txt is a plain-text file at the root of a host that tells crawlers which paths they may request. It is grouped into blocks: a "User-agent" line names a crawler (or "*" for all), followed by "Allow" and "Disallow" rules. A crawler reads the file before fetching anything else and applies the block that matches its name.
The critical limitation: Disallow controls crawling, not indexing. If another site links to a URL you have disallowed, search engines can still list that URL — usually with no snippet, since they were not allowed to read it. To keep a page out of results you need a "noindex" meta tag or X-Robots-Tag header, and the page must NOT be blocked in robots.txt, or the crawler can never see the noindex. For anything truly private, use authentication.
Rule matching is by specificity, not order. Google and Bing pick the longest matching path, so "Disallow: /files/" together with "Allow: /files/public/" blocks the folder but frees the sub-folder regardless of which line comes first. An empty "Disallow:" means nothing is blocked; "Disallow: /" blocks the entire host. Wildcards ("*") and end-of-URL anchors ("$") are supported by the major engines.
Two practical notes. Crawl-delay is ignored by Googlebot — set the crawl rate in Search Console — though Bing and others honour it. And you can add blocks for AI training and answer-engine crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot and more); the well-known operators state they respect robots.txt, but compliance is voluntary, not enforced. Always include a "Sitemap:" line with the absolute URL of your sitemap.
The file is built entirely in your browser. Serve the result at https://yourdomain.com/robots.txt — it applies only to the exact host and protocol it is served from.
Key formulas (reference)
User-agent: * Disallow: /admin/ Allow: /admin/public/ User-agent: GPTBot Disallow: / Sitemap: https://example.com/sitemap.xml
Related tools
These free tools pair well with this page — open them in a new tab to finish your workflow.
Frequently Asked Questions
Where does the robots.txt file go?
At the root of the host: https://example.com/robots.txt. It only governs the exact host and protocol it is served from, so a subdomain or the http version needs its own file. Crawlers request it before crawling anything else on the site.
Does Disallow actually hide a page?
No. robots.txt asks compliant crawlers not to fetch a path, but a disallowed URL can still appear in search results if other pages link to it (usually with no description). To keep a page out of the index, use a noindex meta tag or HTTP header, or require authentication — and do not also block it in robots.txt, or the crawler can never see the noindex.
What does an empty Disallow mean?
"Disallow:" with nothing after it means "nothing is disallowed" — i.e. allow the whole site. "Disallow: /" with a slash blocks everything. This tool emits the empty form for its allow-everything preset.
How do Allow and Disallow interact?
Google and Bing use the most specific matching rule, not rule order. "Disallow: /admin/" plus "Allow: /admin/public/" blocks the admin area but permits the public sub-folder. Longer path matches win over shorter ones.
Does Crawl-delay work with Google?
No. Googlebot ignores Crawl-delay; set the crawl rate in Google Search Console instead. Bing, Yandex and some other crawlers do honour it. The directive is included here because non-Google crawlers still read it.
Can I block AI training crawlers?
You can add Disallow rules for their user-agents — GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot and others — which the "Block AI crawlers" preset does. Compliance is voluntary; well-known operators state that they honour robots.txt, but it is not enforced technically.
Should I list my sitemap here?
Yes. A "Sitemap:" line with the absolute URL of your sitemap helps every crawler discover it, and it is independent of the User-agent blocks. You can list more than one sitemap line.