robots.txt generator

Create a robots.txt to control search crawlers.

Default policy Sitemap URL

Paths to block (one per line, e.g. /admin/)

Save the generated text as a robots.txt file in your site root.

A crawler works through your admin screens, your internal search results and a few thousand filter combinations, and your server bill notices before you do. robots.txt is the file that tells well-behaved crawlers which paths to skip. Choose a policy, list the paths, and this tool writes the file — but read the section on what robots.txt cannot do before you rely on it.

How it works

Choose a default policy

Allow all crawlers writes an open file with an empty Disallow, which is what most public sites want. Block all crawlers writes Disallow: / — use it on a staging or private site only. Custom keeps the site open but excludes the paths you list.

List the paths to exclude

Enter one path per line in Paths to block, each starting with a slash. Matching is by prefix from the start of the URL path and it is case-sensitive, so the trailing slash matters:

Rule	Blocks	Does not block
`/admin/`	`/admin/users`	`/administration`
`/admin`	`/admin/users`, `/administration`	`/user/admin`

Typical candidates are admin areas, cart and checkout pages, internal search result URLs, and endless faceted-filter combinations.

Add the sitemap

Sitemap URL takes the absolute address of your XML sitemap. It is independent of the crawler rules and simply points crawlers at your page list.

The limit that catches people out

robots.txt controls crawling, not indexing. A blocked URL can still appear in search results if other sites link to it — listed without a snippet, because the crawler was never allowed to read it. Worse, blocking a page means the crawler never sees a noindex tag on it, so blocking is exactly the wrong way to remove something already indexed. To take a page out of results, leave it crawlable and put <meta name="robots" content="noindex"> on it, or put it behind authentication. Never do both at once.

Finally, remember the file is public. Anyone can read yourdomain.com/robots.txt, so listing /secret-backup/ advertises it rather than hiding it.

Terms explained

User-agent: The crawler a block of rules applies to. An asterisk means every crawler that does not have a more specific block of its own.
Disallow: A path prefix crawlers are asked not to request. An empty Disallow value means nothing is blocked.
Allow: An exception that reopens a path inside a blocked directory — for example permitting /admin/public/ while /admin/ stays closed.
Sitemap: An absolute URL pointing at your XML sitemap. It sits outside the user-agent blocks and can appear anywhere in the file.
Crawl-delay: A non-standard directive asking for a pause between requests. Google ignores it; some other crawlers respect it.

Frequently asked questions

Where does the file have to go?

At the root of the domain, so it loads at https://yourdomain.com/robots.txt. A copy in a subfolder is ignored, and every subdomain needs its own file.

Will blocking a page remove it from Google?

No. robots.txt stops crawling, not indexing. A blocked URL can still be listed if other pages link to it, and because the crawler cannot open the page it will never see a noindex tag inside. To remove a page from results, allow crawling and add a noindex meta tag, or require a login.

Can I hide private files this way?

Not safely. The file is publicly readable, so naming /admin/ or /backups/ in it tells anyone curious exactly where to look. Protect sensitive paths with authentication and server permissions; use robots.txt only for crawl management.

Should I list my sitemap in robots.txt?

It is optional but costs nothing and helps crawlers that have not been told about the sitemap elsewhere. Submitting it through Google Search Console or Bing Webmaster Tools remains the more reliable route.

Do all crawlers obey robots.txt?

Only the well-behaved ones. It is a voluntary convention with no enforcement, so scrapers and malicious bots ignore it entirely. Anything that must not be fetched needs a real access control.