What llms.txt is and whether you need one

The problem you are actually trying to solve

Someone asks ChatGPT or Perplexity which tool to use for the job you do. Your name does not come up. You go looking for the lever, and a dozen posts tell you to add an llms.txt file. It sounds like the AI version of robots.txt, which is a real standard with real teeth, so it feels like a required step.

It is not the same kind of thing. Understanding the difference saves you from spending a week on a file and then wondering why nothing changed.

What the file actually contains

The specification at llmstxt.org asks for a markdown file with a small, fixed shape:

1. An H1 with the name of the site or project. This is the only required element.
2. An optional blockquote with a short summary of what the site is.
3. Optional markdown paragraphs or lists with more detail, with no headings in them.
4. Zero or more H2 sections. Each one holds a markdown list of links in the form [Name](url): short description.
5. An optional final H2 section called Optional. Links in that section can be skipped when a shorter context is needed.

That is the whole format. The reference implementation and the discussion live in the AnswerDotAI/llms-txt repository on GitHub, and Howard's original write-up is dated 3 September 2024.

Why this step and not another

Here is the honest comparison of the three root files people confuse. Everything in this table is checkable from the linked primary source.

Two of those are honoured by named crawlers with published documentation. The third is a convention that a site owner can adopt and an engine may or may not use.

What we checked, and what we found

On 03 October 2026 we read the public crawler and bot documentation of four companies and looked for any mention of /llms.txt:

• OpenAI bots documentation
• Google Search Central, overview of Google crawlers
• Anthropic support article on web crawling
• Perplexity bots guide

None of those four pages stated that the company's crawler or answer system reads /llms.txt. That is not proof that no system ever fetches it. It does mean that, today, nobody has promised you anything. Re-check the four links yourself before you budget time.

So why add it at all

Three reasons that survive the honesty test:

• It is cheap. A small site is an hour of work, once.
• It forces you to name your ten best pages in one line each. That exercise is useful on its own, and the descriptions often improve your page titles and meta descriptions.
• Some documentation platforms and developer tools already consume it. If your buyers are developers reading docs through an assistant, the file can be read directly when someone pastes the URL.

What it will not do: it will not make an engine recommend you, and it will not substitute for other sites naming you. In our experience the answers engines give are built mostly from pages that third parties publish and cite. Your own root file is a convenience, not a vote.

How to do it

Allow sixty to ninety minutes for a site with fewer than fifty pages.

1. Open a plain text editor. Do not use a word processor, which will insert curly quotes.
2. Write an H1 with your site or product name: # Harbour Billing.
3. Add one blockquote line that says what you do and for whom. Keep it under thirty words.
4. List the ten to thirty URLs that actually answer buyer questions: pricing, docs, integrations, security, support, contact, the two or three guides people read before they buy.
5. Group them under H2 headings such as ## Docs, ## Product, ## Company.
6. Write each link as [Page name](https://full-url): one line on what is in it. Use absolute URLs, not relative ones.
7. Move anything nice-to-have (archive posts, legal pages) under a final ## Optional heading.
8. Save as UTF-8 with the exact name llms.txt, all lower case.
9. Upload it so it is served at https://yourdomain.com/llms.txt, at the root, not in a subfolder. Serve it as text/plain or text/markdown.
10. Confirm your robots.txt does not block /llms.txt itself.
11. Confirm every page you listed returns HTTP 200, renders without JavaScript, and is not disallowed in robots.txt. A beautiful index pointing at blocked pages is worth nothing.
12. Write today's date inside the file as a comment line, and put a calendar reminder to regenerate it whenever you ship or retire pages.

A worked example you can repeat

This is a made-up company, used only to show the shape of the file.

Then run two checks from a terminal:

The first should return 200 and a text content type. The second should print your file, not an HTML error page dressed up as a 200.

How to tell it worked

Separate three questions that people usually mash together.

1. Is the file readable?

Check your server logs for requests to /llms.txt and note the user agent string. Named agents to look for include GPTBot, ClaudeBot and PerplexityBot, all documented in the links above. Note one detail that trips people up: Google-Extended is a product token you use inside robots.txt to control use of your content, not a separate crawler that shows up in logs with that name.

2. Does an engine use it when pointed at it?

Paste the URL into ChatGPT, Claude or Perplexity and ask it to list the pages. If it can summarise the file, it can read it. This proves fetchability. It proves nothing about unprompted answers.

3. Did unprompted answers change?

This is the only test that matters commercially, and it is the one most people skip. Do it like this:

1. Write twenty questions a buyer would ask that never contain your brand name. For example: "best GST invoicing software for a 10 person agency in India".
2. Ask all twenty across more than one app, because they answer the same question differently on the same day from different sources.
3. Count how often your business is named, and record the date.
4. Publish the llms.txt file. Change nothing else that week.
5. Repeat the same twenty questions 30 days later.
6. Compare the two mention rates with their ranges. Twenty questions is a small sample, so a jump from 2 out of 20 to 4 out of 20 is noise, not a result.

If you want the longer sequence this test sits inside, see /blog/what-to-do-in-the-first-30-days.

Where AI Knows Us fits

AI Knows Us measures across ChatGPT, Gemini, Perplexity, Claude and Google AI Overviews. Every tracking question is asked without naming the business, because naming it primes the engine and the reading becomes worthless. Every rate is shown with a 95 percent range, and two numbers whose ranges overlap are reported as not having moved, which is exactly the discipline the llms.txt before-and-after test needs. The product also does the fixing as well as the measuring: it writes pages, publishes them and asks other sites to name the business. Thirteen agents run that work: intake, discovery, analyst, diagnosis, planner, writer, editor, publisher, scout, outreach, experiment, lifecycle and the copilot. It is made by Clyra Labs in India. Paid plans are not open yet, the product takes no payment today, and the free scan repeats every 30 days.

Other tools measure AI answer visibility too, including Profound, Peec AI, Otterly.AI and Scrunch AI. Compare their current feature lists and pricing on their own sites before choosing. We do not publish their prices here because they change and we will not restate them second hand.

Who should not choose AI Knows Us

• You only want a file generator. An llms.txt builder is a five minute script, and you do not need a measurement product for it.
• You need a signed contract and an invoice today. Paid plans are not open yet.
• You want brand-name prompt tracking only, where the question contains your name. That is not how the tracking questions here are written.
• You want technical SEO auditing, site speed or schema fixes as the main service. That is a different category and other tools do it better.

If you are deciding what an outside team should deliver each month, # covers scope and hours.

FAQs