What a competitor's robots.txt tells you, and the four things it cannot

A competitor's robots.txt is a published document on a fixed path that anybody can read without permission, a login or a tool, and almost nobody reads it. On 20 October 2026 we fetched the file from 26 domains our own citation audits had named, followed every redirect, and parsed each one into the rule group that applies to each of fourteen named agents. 24 of the 26 were readable. What came back settles three useful things about a rival and leaves four open, and knowing which is which is the difference between competitive intelligence and a guess with a source.

The three it settles: whether that company has made any decision about AI agents at all, whether the decision was taken per operator or as a single switch, and which parts of the site it does not want crawled. Across our 24 files the answer to the first question was mostly no. Not one refuses GPTBot, OAI-SearchBot or OAI-AdsBot at the root, and the only host turning assistant agents away in its file is a single news publisher. The full per agent and per host counts are published as 26 robots files with every rule counted.

The three readings a rival's file gives you

The second row is where the value sits. A file that names agents individually and gives them different answers was written by somebody with a position, and that position is commercial information about how the company sees its own content. The one host in our sample doing it is a publisher whose archive is inventory, and the reading is set out in the one news publisher that blocks eleven of fourteen agents.

The four things the file cannot tell you

• Whether a request actually gets through. A file is a request to well behaved crawlers and a server is what answers. In this same sample one of the most cited hosts in the category permits all fourteen agents in its file and returns HTTP 403 at the edge to five named crawlers and to a request with no agent string at all, confirmed on two readings hours apart.
• Whether the rival is being quoted. Permission and selection are separate measurements with separate denominators. Our own domain permits every agent and appeared in the source list of 6 readings of 1,260 on 16 October 2026.
• Why a rule is there. A named block can be a licensing position, a response to one abusive crawl, or a line copied from a template in 2023 that nobody has read since. The file records the rule and never the reason.
• Anything about deeper paths unless you ask. Our counts describe the root. A host can permit the root and refuse a directory, and a reading that does not say which path it measured is not a reading.

Building a host list, then fetching it properly

Twenty six hosts took under an hour including the mistakes. The order below is the one that survived them.

• Build the list from your own data rather than from a market map. Ours was the ten most cited domains of our second audit, the six named specialist competitors, the adjacent suites and directories our rows surfaced, four small agencies the audit named as local winners, and our own site. A file only tells you something once you know why that host is on the list.
• Fetch with redirect following on. Our own first pass without it obtained 15 of 26 and recorded eleven hosts as unreadable. Every one of those eleven was a hop between a bare domain and a www name or between HTTP and HTTPS, and none of them was refusing anything.
• Parse per agent, never by eye. An agent is matched to one group, the most specific one that names it, and the wildcard group then stops applying to it. A file read top to bottom gives a plausible answer and the wrong one on exactly the files where somebody made a decision.
• Separate unreadable from unrestricted. Two of our 26 returned HTTP 403 to the request for the file and are reported as not usable rather than as blocks, because a 403 from one of those hosts has already turned out in this programme to be a rate limiter with an HTTP 429 behind it.
• Save the raw bytes and the final address. The rules apply to the host, protocol and port that served the file, and whether a non 200 was a refusal or a browser challenge is answerable later only if you kept the body.
• Put the date beside every row, because these are documents their owners can edit the same afternoon.

The other public records worth reading in the same hour

A robots file is the cheapest of a small set of things a competitor publishes whether they mean to or not, and they are more useful read together than separately.

• The sitemap the robots file points at, which gives you the page inventory and often the publishing cadence.
• The llms.txt file if there is one, and its absence is itself a reading.
• What the first HTTP response actually contains for a pricing or specification page, because a figure written in by a script is not a figure an agent that fetches the document has.
• The entity record a search engine holds about the company, which is a separate public artefact and is worth looking up for your own organisation as well: the method is in what the knowledge graph record holds and how to look up your own entity.

What our 26 host reading does not support

It is 26 domains on one date, chosen because our own audits named them, which makes it a reading about one category rather than about the web. A news archive, a research library or a stock photography business would sort differently, and the way to find that out is to read the files for the hosts that turn up in your own readings rather than ours. No paywall, cookie banner or consent wall was measured anywhere in it. The citation figures beside it come from two engines, ChatGPT and Gemini, while our product watches five surfaces, so Perplexity, Claude and Google AI Overviews are unmeasured in those counts.

Reading a rival's permission file: the questions that follow

Is reading somebody else's robots.txt allowed?

It is a public file served to every client that asks, on a fixed path, which is the whole reason it exists. Fetching it is an ordinary request and the only courtesy worth observing is not hammering a host with a loop.

What does it mean when a competitor names no AI agent at all?

That nobody there has made a decision yet, and the wildcard group is deciding for them. In our sample that was the normal condition, and it means the next question about that rival is not about permission at all.

Can I tell from the file whether a rival is being cited by assistants?

No, and the two records have to stay separate. Access decides whether a page can be fetched. Being quoted turns on whether the page carries something specific enough to answer a question, and those two fail for different reasons and are fixed by different work.

Why did two of the 26 come back unreadable?

Both returned HTTP 403 to the request for the file itself. They are reported as not usable rather than as blocks, because a forbidden response under an automated request needs a second reading in a separate session before it describes a policy, and one of those two hosts has already produced one that was a rate limiter.

How often should the list be re read?

Put it on the same schedule as your own release notes, because a robots file changes when somebody ships. Our own copy of this reading carries 20 October 2026 for that reason, and the citation run that chose the 26 hosts is published as 597 readings across ChatGPT and Gemini in three markets.

One request, and most of what you learn is what to stop worrying about

Read twenty six of these and the useful result is negative: the permission layer across the sites already winning this category is wide open, so an absence from answers is not being caused by anybody's file. That moves the first hour of work off the door and onto the room, which is where the writing is. A rival who has made no decision about any of this is not withholding anything from you, and that is the most common thing twenty six files will tell you. The hour it gives back goes into the pages, which is the half of this we are hired for, across ChatGPT, Gemini, Perplexity, Claude and Google AI Overviews. What a reading of ours is allowed to claim, with each window and each pull date printed beside it, is on our proof page.