Four robots.txt cases a parser has to get right before it reads anybody else's file

Before our parser read a single real file on 20 October 2026 it had to sort four small files whose correct answer was already written down: one that blocks a named agent, one that allows a named agent inside an otherwise closed site, one group header naming two agents at once, and a file with no named group at all so the wildcard decides. If any of the four comes out wrong the parser stops and the run does not happen. That is the whole safeguard, and it exists because getting any one of those four wrong does not produce an error. It produces a full table of plausible rows, every one of them wrong in the same direction.

A robots file looks like a list of instructions and behaves like a lookup. Under the Robots Exclusion Protocol an agent is matched to exactly one group, the most specific one that names it, and once that group applies the wildcard group stops applying to that agent entirely. So a parser that reads the file the way a person reads it, top to bottom, accumulating rules, will produce an answer for almost every real file and the answer will be wrong on precisely the files where somebody has made a decision. Our counted reading of 24 readable files is published as 26 robots files with every rule counted, and this is the gate it had to pass first.

The four fixtures, and what each one catches

The second and third are the ones that bite in practice. A named allow inside a closed site is how a publisher lets one operator in, and it is the exact configuration we found on the one news publisher in our sample. Stacked user agent lines are how most hand written files express a shared policy, and a parser that handles only one name per group silently halves the rules it applies.

Why a parser failure looks like a result

Each of the four errors above changes every row of the output in a consistent direction, which is what makes it so hard to notice. If named groups are being ignored, every file reads as its wildcard, so a sample of hosts that mostly allow everything comes back as a sample that allows everything, and the two or three files where somebody made a real decision vanish into the pattern. The output has no blanks in it and no warnings beside it.

This is the same failure shape that has produced four separate wrong answers in this programme. A classifier whose pattern required a literal string sorted an operator's own documentation into vendor marketing. A geo parameter was accepted by an interface, returned success and did nothing. A similarity threshold was set above the highest score its data could produce, so nothing was ever flagged. The general question worth asking of any check is what it would report if it were handed nothing, and the version of that discipline written for page checks is asserting what a page must contain instead of comparing two fetches.

Building the four fixtures yourself

Each is a handful of lines in a text file, and the expected answer is one word per agent. Write the answer down before you run anything, because a fixture whose answer you decide after seeing the output is a fixture that agrees with your code.

• Keep them in version control beside the parser. A fixture that lives in somebody's notes stops being run within a week.
• Assert on the decision, not on the parse tree. The question is allowed or blocked for this agent at this path. A test that checks the shape of the data structure passes happily while the decision is wrong.
• Make a failure stop the run. Not a warning in a log. Our parser exits and prints which fixture failed, in the same way our source classifier refuses to load when any of 24 named examples sorts wrongly.
• Add a fixture every time a real file surprises you. The four above are the floor. A file with a group naming an agent that appears nowhere else, or a disallow with an empty value, becomes the fifth and sixth.
• Handle the fields separately from the rules. A sitemap line is not a rule and must not end up inside a group's disallow set, and the related question of keeping that reference current is covered in why a sitemap is a feed rather than a file.

The three things to record beside every parsed row

A decision without its provenance cannot be rechecked, and a robots reading exists to be rechecked.

• Which group the agent matched, named or wildcard. Two agents reading as blocked for different reasons are two different facts about the publisher.
• The path the decision was taken at. Our counts describe the root. A file can permit the root and refuse a directory, and a count that does not say which path it measured is not a count.
• The date the file was read, and the final address it was read from. These are files their owners can change this afternoon.

What a verified parser still does not give you

It gives you what the file says. It does not give you what the server does, and in this dataset those two disagree on one of the most cited hosts in the category: a help centre that permits all fourteen agents we tested in its robots file returns HTTP 403 at the edge to five named AI crawlers and to a request carrying no agent string at all, confirmed on two readings taken hours apart. A parser reading only the file would have recorded that host as fully open. Reading both layers is a separate exercise and the protocol for it is published as a dated test of robots.txt, HTTP access and named agents.

Group matching, fixtures and missing files: the awkward parts

Why does the most specific group win rather than all matching rules applying?

Because the protocol assigns each agent to one group. A file naming an agent is making a decision about that agent, and the wildcard group is what applies to everybody the file has not named. That is why adding a named group with a single allow line can quietly open a directory the wildcard group was closing.

What goes wrong with two user agent lines above one set of rules?

Nothing, if the parser attaches the rules to both names. A parser that keeps only the last name it saw leaves the first agent with no group, which pushes it into the wildcard and reports a decision the publisher did not make. It is the commonest hand written shape in real files, which is why it is one of the four.

Should a parser treat a missing file as permission?

A 404 is a host publishing no rules, and a well behaved crawler proceeds. An unobtained file is different: a 403, a timeout or a redirect you declined to follow leaves you not knowing. Our own reading separates the two and reports 24 readable of 26 fetched rather than collapsing them.

Where does the fixture set stop growing?

Four is the floor because each one catches a different class of error, and the set grows from the real files. Every time a published file produces a row that surprises whoever reads the output, that file becomes a fixture with its answer written down, and the parser is rerun against the whole set.

Is a verified parse any sort of signal about getting quoted?

No, and the two records belong in separate columns. Our own measured citation figure is 3 readings of 597 on 15 October 2026 and 6 of 1,260 on 16 October 2026, both on ChatGPT and Gemini, and neither of those numbers has anything to do with a robots file. The method behind the first is published as 597 readings across ChatGPT and Gemini in three markets.

A fixture file is cheaper than a correction

Four tiny files and twenty minutes stand between a parser and a table of other people's decisions that is confidently wrong. We publish the counts that parser produced, and the part we are actually paid for is the measurement behind them being reproducible by the person reading it: the two Google Search Console exports underneath our own claims, with each window and the day each file was pulled, are printed on our proof page for the same reason. A number nobody can rebuild is a number somebody has to take on trust, and this category has enough of those already.