How to write a robots.txt, with copyable examples
The four lines most sites need, the rules Google publishes, recipes for the common cases, and the three mistakes that cost real traffic.
Published by AI Knows Us · Updated 13 October 2026
A robots.txt is a plain text file at the root of your domain that tells crawlers which paths they may fetch. Most sites need four lines, and you can write them faster than a generator can. Here is the version that suits the majority of sites:
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlSave that as robots.txt, upload it to the root so it answers at https://example.com/robots.txt, and swap in your own sitemap URL. The rest of this page covers the rules, the copyable recipes for common cases, and the three mistakes that cause real damage.
The one distinction that matters
Robots.txt controls crawling. It does not control indexing. Google's documentation is explicit: for a noindex rule to work, "the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler", because "if the page is blocked by a robots.txt file or the crawler can't access the page, the crawler will never see the noindex rule, and the page can still appear in search results, for example if other pages link to it".
So use robots.txt to save crawl effort on paths nobody needs, and use a noindex meta tag on pages you want kept out of search results, leaving those pages crawlable so the tag is read. Treat the file as public, since anyone can read it, and keep private paths out of it.
The rules Google follows
| Rule | Detail |
|---|---|
| Supported fields | user-agent, allow, disallow and sitemap. Google states that other fields, including crawl-delay, are not supported |
| Location | The root of the host. Rules apply only to the host, protocol and port serving the file, so each subdomain needs its own |
| Wildcards | * matches zero or more characters, $ matches the end of the URL |
| Case | Paths are case sensitive and must start with / |
| Conflicts | The most specific rule wins, measured by the length of the rule path. Where rules of equal length conflict, Google uses the least restrictive |
| Size | Google reads up to 500 kibibytes and ignores anything beyond that |
| Caching | Generally cached for up to 24 hours, so allow a day for a change to take effect |
Copyable recipes
Block a folder and a single file. The trailing slash keeps the rule to that directory.
User-agent: *
Disallow: /internal-tools/
Disallow: /private/invoice-template.pdf
Sitemap: https://example.com/sitemap.xmlAllow one file inside a blocked folder. The longer Allow rule wins because it is more specific.
User-agent: *
Disallow: /reports/
Allow: /reports/public-summary.htmlStop crawling of search and filter URLs. This is the most useful rule on an ecommerce site, where filter combinations generate large numbers of near identical URLs.
User-agent: *
Disallow: /*?s=
Disallow: /*?sort=
Disallow: /*&filter=
Disallow: /search/Block one file type everywhere. The $ anchors the match to the end of the URL.
User-agent: *
Disallow: /*.sql$
Disallow: /*.log$Block an entire staging or development site. Put this on the staging host only, and never let it reach production.
User-agent: *
Disallow: /Allow search assistants while declining training use. These tokens are published by the companies that run the crawlers. OpenAI documents GPTBot for content that may be used to train its models and OAI-SearchBot for surfacing sites in ChatGPT search. Anthropic documents ClaudeBot for collecting content that may contribute to training and Claude-SearchBot for improving search results. Google documents Google-Extended as the token for whether content may be used to train Gemini models, and states that it does not affect inclusion in Google Search. Perplexity documents PerplexityBot for surfacing and linking sites in its results.
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlDecide that one deliberately. Blocking the search tokens removes you from the answers those products give, which for most businesses is a larger cost than the training question. Publishers selling access to their archive often decide the other way, and both positions are reasonable for their own reasons.
Three mistakes that cause real damage
A staging rule shipped to production. Disallow: / on a live site stops crawling of everything. This is the single most expensive line in SEO, and it reaches production during releases more often than anyone expects. Add a check for it to your deploy checklist.
Blocking CSS and JavaScript. Google renders your pages, so blocking /assets/ or /wp-includes/ can leave it seeing a broken layout. Leave your stylesheets and scripts crawlable.
Using Disallow to hide a page from search results. A blocked page can still be listed if other sites link to it, because the crawler never reads your noindex tag. Keep it crawlable and use the tag.
How to test it
- Open
https://yourdomain.com/robots.txtin a browser. It should return the file, not a 404 and not your home page. - Check the robots.txt report in Search Console, which shows the files Google found for your top hosts, when it last fetched them, the file size, and any parse errors.
- Use the URL Inspection tool on a specific page to confirm Google can fetch it.
- Serve the file as
text/plainand in UTF-8, with one directive per line. - Allow up to a day for a change to take effect, since Google generally caches the file for 24 hours.
What this page covers, and what it leaves out
This page covers writing, testing and maintaining robots.txt, with the rules Google publishes and recipes for common cases. For keeping pages out of search results, use the noindex guidance instead. For sitemaps and canonical tags, use a dedicated guide for each.
The write a robots.txt questions worth an answer
Do I need a robots.txt file at all?
A site with nothing to block works fine without one, and crawlers treat a missing file as permission to crawl everything. Publish the four line version anyway, because it declares your sitemap and gives you one obvious place to add a rule later.
Does robots.txt stop a page appearing in Google?
No, and that is the most useful fact on this page. Blocking a path stops the fetch, and a blocked URL can still be listed if other sites link to it. Use a noindex meta tag on a crawlable page when you want it out of the results.
Is crawl-delay supported?
Google states that it does not support crawl-delay. Some other crawlers do honour it, so include it only for those and control Google's crawl rate through server capacity and Search Console instead.
Do subdomains need their own file?
Yes. The rules apply only to the host, protocol and port serving the file, so blog.example.com needs its own robots.txt. This catches people out on marketing subdomains and help centres.
Should I use a robots.txt generator?
A generator is a convenient starting point, and the file is short enough to write directly from the recipes above. Whichever you use, read every line before it ships, because one wrong Disallow outweighs every other setting on the site.
The other robots your file may be addressing
A correct robots.txt controls what search crawlers fetch. Assistants send their own crawlers, with their own names, and your file either welcomes them, blocks them or says nothing. A blanket rule written years ago can quietly cover them.
Read your file against the agent names each assistant publishes, and decide on purpose. Then ask each assistant about a page you meant to be open and see whether it can describe it. AI Knows Us checks what the answers say from the buyer's side.