Robots file generator: the settings it asks for, the output to expect, and how to verify it before it goes live

A robots file generator turns a few form answers into a robots.txt file: which crawlers you address, which paths you keep them out of, your crawl delay preference if any, and the location of your sitemap. It is a formatting aid rather than a decision maker. The generator writes correct syntax, and it does not know which of your URLs matter, so the two steps that decide whether the file helps or hurts are choosing what to disallow and verifying the result against your real URLs before you publish it. A generated file that blocks a directory your product pages live under will quietly remove those pages from search. Generate, verify, then publish, in that order, and the file is a five minute job.

This page covers generating and verifying robots.txt. For deciding which AI crawlers to allow, see a page on crawler access policy.

What robots.txt controls and what it does not

The file sits at the root of your domain, at /robots.txt, and tells cooperating crawlers which paths they may request. It controls crawling. It is not a way to keep a page out of search results, because a URL that is linked elsewhere can still appear without being crawled, and it is not a security measure, because the file is public and names the paths you want left alone. Use noindex in a meta tag or an HTTP header to keep a page out of the index, and use authentication for anything private. Keep both facts in mind while you fill in a generator form, because that is where most generated files go wrong.

The five questions every generator asks

• Which user agents. The safe default is one group addressed to all crawlers with the wildcard, plus a named group only where you genuinely want different rules for a specific crawler.
• What to disallow. Admin and login paths, internal search result pages, cart and checkout, thank you pages, staging subfolders, and faceted filter parameters that produce endless combinations.
• What to explicitly allow. Needed when you disallow a folder but want one file inside it crawled. Allow is also how you keep CSS and JavaScript reachable, which matters because a crawler that cannot load your stylesheet cannot see your page as a visitor does.
• Crawl delay. Google ignores crawl delay and uses its own rate, which you adjust in Search Console if a server is struggling. Some other crawlers honour it.
• Sitemap location. Give the full absolute URL. This line is independent of any user agent group and is the single most useful thing in the file.

A clean generated file, annotated

This is what a good generator produces for a standard site. Replace the domain and the paths with your own.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cart/
Disallow: /checkout/
Disallow: /search
Disallow: /*?orderby=
Disallow: /*?filter_

Sitemap: https://example.com/sitemap.xml

Four things make this file correct. Each directive sits on its own line. The paths start with a slash and are matched as prefixes, so /search also covers /search/anything. The wildcard patterns target query strings that create duplicate listings. The sitemap line is absolute and sits outside the group.

Verify before you publish

Run these six checks in order. They take a few minutes and they catch everything that matters.

Check	How	What right looks like
Location	Request yourdomain.com/robots.txt in a browser	Plain text at the root, HTTP 200, served over your canonical domain
Encoding	View the raw file	UTF-8 plain text, no HTML wrapper, no smart quotes pasted in from a document
Your money URLs	Test ten real URLs you need crawled against the file	All ten allowed
Your blocked URLs	Test the paths you intended to block	Blocked, and nothing adjacent blocked by accident
Assets	Test one CSS file and one JavaScript bundle	Allowed, so pages render for the crawler
Search Console	Open the robots.txt report and the URL Inspection tool	Google reports the file as fetched, and inspection on a key URL shows crawl allowed

Keep a copy of the previous file before you replace it. Rolling back in thirty seconds is the difference between a mistake and an incident.

The mistakes generated files commonly contain

• A blanket disallow left in from a staging configuration. A single line reading Disallow: / blocks the entire site. This is the most expensive two characters in technical SEO, and it usually arrives when a development robots.txt is copied to production.
• Blocking a parent folder that contains live pages. Disallow: /products/ to stop filter URLs also stops every product.
• Trying to remove a page from Google with Disallow. A disallowed URL cannot be recrawled, so a noindex tag on it is never read, and the URL can persist in results. Allow the crawl, serve noindex, and remove the disallow only after the page has dropped out.
• Blocking CSS and JavaScript. Generators that ask you to block /assets/ or /static/ encourage this. Keep rendering resources open.
• Relying on robots.txt for privacy. Listing /internal-reports/ in a public file advertises where the reports are.
• One file on a subdomain applied to another. Each host needs its own file, so shop.example.com and example.com are separate.
• Case and trailing slash errors. Paths are case sensitive, and /Blog does not match /blog.

When to write it by hand instead

Use a generator for a first file on a straightforward site. Write it by hand, or get a developer to, when you have multiple sitemaps, several user agent groups with different rules, a large faceted catalogue where parameter handling needs care, or a migration where crawl budget is being steered deliberately. In those cases the file carries real decisions and deserves a review and a version history alongside the rest of your code.

On WordPress, Shopify and most hosted platforms the file is generated dynamically and edited through a plugin or theme setting rather than uploaded. Check how your platform serves it before you try to upload a file by FTP, because an uploaded file can be ignored in favour of the platform's own.

Where does the robots.txt file go?

At the root of each host, reachable at https://yourdomain.com/robots.txt. A file in a subfolder is not read. If your site answers on both www and non www, make sure the canonical host serves the file and the other redirects to it.

Do I need a robots.txt file at all?

A small site with nothing to block functions without one, and crawlers treat a missing file as permission to crawl everything. It is still worth having, because the sitemap line helps discovery and the file gives you one place to act when something needs blocking later.

Is a free robots.txt generator safe to use?

Yes, since the output is a short text file you read before publishing. Paste the result into a text editor, read every line, check it against the six verification steps above, then publish. Never publish generator output unread.

How do I block one page rather than a folder?

Give the exact path, for example Disallow: /old-offer-page. To keep it out of search results instead of out of crawling, leave it crawlable and serve a noindex meta tag on that page.

How quickly does a change take effect?

Google refetches robots.txt regularly and typically picks up changes within a day. After a significant change, open the robots.txt report in Search Console to confirm the version Google holds matches the one you published.

The file is read by more bots than the ones in your template

Your generator offers presets for Googlebot, Bingbot and a few others. The file is now also read by the crawlers that feed AI assistants, and each of them has its own user agent name that a preset may not list.

Decide what you want from each, then look at the outcome: ask an assistant about your site and see whether it knows the pages you opened. AI Knows Us runs that question across several assistants, so you can compare the file's intent with what comes back.