How to log and monitor AI crawler visits: identifying the agents, verifying them against published address ranges, and the counts to keep

You cannot identify an AI crawler from the name it gives you, because a user agent string is a line of text anybody can send. You identify it by matching the source address against the ranges the operator publishes, and on 16 October 2026 we fetched all four of OpenAI's published range files and counted them: GPTBot 18 ranges, OAI-SearchBot 39, ChatGPT-User 234, OAI-AdsBot 2. Every one of those 293 entries is IPv4 and not one is IPv6, while Google's published Googlebot file held 317 ranges of which 147 are IPv6. Those two facts decide how log analysis for AI crawlers has to be set up, and they are the reason it is a different job from reading your logs for Googlebot.

Why the agent string is evidence of nothing on its own

A test we ran the same day makes that concrete. We knocked on 16 public doors under 11 different names, and 23 of the 176 attempts were refused outright, with the refusals turning entirely on the name given: one site welcomed every bot name and shut out desktop Chrome, another shut out every name and answered only the request that gave none. Edge rules act on that line of text, which means anything reading your logs can be fooled by it.

So a log analysis counting agent strings is counting claims, and one counting verified addresses is counting visits. Keep both columns and never merge them. The reverse DNS and published range methods for Google's own crawler are in our guide to what Googlebot is and how to verify a request; the assistant crawlers publish their data in a different shape.

The agents to search for, named from the operator's own documentation

OpenAI's crawler documentation, read on 16 October 2026, names four user agents rather than the three most lists carry.

• GPTBot, which the documentation describes as used to crawl content that may be used in training generative AI foundation models.
• OAI-SearchBot, for search. The documentation states that sites opted out of it "will not be shown in ChatGPT search answers, though can still appear as navigational links".
• ChatGPT-User, for actions a person triggers. The documentation states plainly that "Because these actions are initiated by a user, robots.txt rules may not apply", and separately that it "is not used to determine whether content may appear in Search".
• OAI-AdsBot, which the documentation describes as visiting only pages submitted as ads, to check them against policy.

Two details on that same page are worth more to a log analyst than the names. The documented example strings carried OAI-SearchBot/1.4, GPTBot/1.4, ChatGPT-User/1.0 and OAI-AdsBot/1.0 on the day we read it, and for two of them the documentation says the version number may change, so a filter pinned to a version will silently stop matching. And when fetching robots.txt, the documentation says OpenAI may add a `robots.txt` marker to the string, so that site owners whose logs omit paths can still tell a robots fetch from a content fetch.

The published address ranges, counted

Each file below was fetched on 16 October 2026 and returned HTTP 200. We parsed the prefixes array in each and counted entries carrying an IPv4 prefix and an IPv6 prefix separately. The creationTime column is the field each file publishes about itself, which tells you how stale your copy is.

file	ranges	IPv4	IPv6	creationTime in the file
openai.com/gptbot.json	18	18	0	22 September 2026
openai.com/searchbot.json	39	39	0	2 January 2026
openai.com/chatgpt-user.json	234	234	0	7 October 2026
openai.com/adsbot.json	2	2	0	12 May 2026
Googlebot ranges, published by Google	317	170	147	15 October 2026
Google special crawlers ranges	272	136	136	15 October 2026

Three readings follow. The fetcher that arrives because one person asked one question publishes 234 ranges, thirteen times the training crawler's 18, so a narrow allowlist built around GPTBot misses most of the traffic that arrives on the day a buyer asks about you. The search crawler's file carried a creationTime from January while the user fetcher's was nine days old, so the two go stale at different rates and refreshing one is not refreshing the other. And because every OpenAI entry is IPv4, a verification rule written only against IPv6 has nothing to match, while the Google equivalent has 147 IPv6 ranges to work with.

The fields to record, and the counts to publish monthly

• Timestamp with time zone, the full agent string unshortened, the path, the HTTP status and the bytes returned. Record the bytes, because a success code sitting on a near empty body is a delivery you will otherwise count as a win.
• The source address, and a separate verified column recording whether it matched a published range on the day you checked.
• Whether that agent had asked for your robots file before it asked for a page, and how long before. The marker described above makes this readable in a log with no paths.
• Whether the request reached origin or was answered at the CDN, because a refusal at the edge may never reach your application log at all.
• Challenges and throttles you issued, kept in their own column rather than lumped in with outright refusals.

Then publish three monthly counts rather than one: verified fetches per agent, unverified requests claiming it, and refusals. Those three move independently, and a single crawler visits number hides which of them changed. When the log shows a crawler arrived and left, the next question is what it got, and the page level checks for that are set out in our eight checks on a single page, in order.

What a log count does not tell you

A verified fetch proves a page was delivered. It does not show what the fetch was used for, and it is not a citation. The documentation itself keeps access and answering separate: a site allowed for both crawlers may be crawled once for both purposes, and a robots file change may take about a day to take effect. So this work is two records kept side by side, your log of verified fetches and a separate record of whether an assistant named you, and the gap between them is the thing worth managing.

Questions about reading AI crawler visits in a server log

Which user agents should I search my British server logs for?

Start with the four OpenAI names above, then add the operators you care about and read each one's own documentation for the current list on the day you build the filter. Search on the operator name rather than a full string with a version in it, because the documentation states the version number may change.

Our logs are full of GPTBot. How much of it is genuine?

Match the source address against openai.com/gptbot.json, which held 18 IPv4 ranges when we fetched it on 16 October 2026. Count a fetch as that crawler's only when the match passes, and keep the unmatched lines in their own column rather than discarding them, because a rise in spoofed requests is its own signal.

We disallowed an agent and it is still in the log. What went wrong?

Three causes are worth checking in order. The fetch may be a user triggered one, and OpenAI's documentation states that for those "robots.txt rules may not apply". The change may be too recent, since the same documentation says it can take about 24 hours for a robots file update to take effect. Or the line is not that crawler at all, which is what the address check settles.

How do I tell a scheduled bulk crawl apart from a single live retrieval?

The agent name does most of it, since the operators give the two jobs different names, and the traffic shape does the rest. Bulk crawling arrives in runs across many paths. A live retrieval arrives alone, on one path, from a far wider address space, 234 ranges against 18 in OpenAI's case.

Why is none of this traffic in our analytics?

These requests never execute your measurement script, so an analytics property has no row for any of them and never will. The server log and the CDN log hold the whole record between them, and it is the CDN log that carries the refusals your application was never told about.

What your logs cannot answer, and the measurement that can

A log tells you who knocked. It cannot tell you whether an assistant named your organisation when a British buyer asked a question in your category, which is the number all this log work is ultimately for. Our own figure for that is 3 citations of our domain in 597 readings on 15 October 2026, and 6 in 1,260 on 16 October from a question set designed separately. Neither is a figure anybody would put in a brochure, and publishing both is the point: two runs, eleven days apart in design, agreeing. Selling that measurement is our business, and AI Knows Us is the product that runs it, which is why the agent names and the range counts on this page are published in a form you can re-run without us. One scope note on those citation figures. They came from questions put to two engines, ChatGPT and Gemini, while our product watches five surfaces, and the gap between those two statements is not something a log can close for you. A log is per crawler, and a crawler is not an assistant.