OpenAI documentation: how it is organised, and the pages that matter for a website owner

OpenAI's developer documentation lives at developers.openai.com/api/docs, and the older platform.openai.com/docs addresses redirect there. It is organised into a handful of tracks: get started and quickstart, core concepts for the API, the model catalogue, guides grouped by capability, agents and tools, production topics such as rate limits and prompt caching, and the API reference. Alongside those sit resource pages that change often and are worth bookmarking individually: the changelog, deprecations, pricing, and the crawler page. If you own a website rather than build on the API, one page matters more than all the rest, and that is the crawler documentation.

This page covers how to navigate the documentation and what the crawler section says. For the wider question of how ChatGPT selects what it cites, see where ChatGPT gets its sources.

The map, by the question you arrived with

What you want	Where it lives
Make a first API call	Quickstart, then the core concepts section
Choose between models	The model catalogue, which lists capabilities and context sizes per model
Exact parameters and response shapes	The API reference, organised by endpoint
Cost of a call	The pricing page, which is the only figure to quote from
Structured output, tool calling, embeddings, audio	The guides, grouped by capability rather than by model
What changed, and what is going away	Changelog and deprecations
Working code to copy	The Cookbook, which is examples rather than specification
Whether a problem is yours	The status page
Whether ChatGPT can read your site	The OpenAI crawlers page

Two habits make the docs much easier to live with. Read the changelog before debugging anything that worked last month, and treat the API reference as the authority when a guide and the reference disagree, because the reference is generated from the interface itself.

The crawler page, read closely

This is the part of the documentation most site owners need and almost none have read. OpenAI documents three separate user agents, and states that each robots.txt setting is independent of the others.

• OAI-SearchBot is for search. It is the crawler used to surface websites in ChatGPT's search features. The documentation states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links, and it recommends allowing OAI-SearchBot in robots.txt along with requests from the published IP ranges at openai.com/searchbot.json.
• GPTBot is the crawler for content that may be used in training OpenAI's generative foundation models. Disallowing GPTBot indicates that a site's content should not be used in that training. Its published IP ranges are at openai.com/gptbot.json.
• ChatGPT-User is the agent used when a user action in ChatGPT or a custom GPT causes a page to be fetched. The documentation says it is not used for automatic crawling of the web, that because the action is user initiated robots.txt rules may not apply, and that it is not used to decide whether content appears in search. Its IP ranges are at openai.com/chatgpt-user.json.

Two consequences follow directly from that text. First, allowing search and declining training are separate decisions, and you can make them differently. Second, OAI-SearchBot is the one that governs appearing in ChatGPT search answers, so a blanket block of every AI user agent takes you out of that surface.

The documentation also notes that it can take around twenty four hours from a robots.txt update for the search systems to adjust, and that where a site allows both bots a single crawl may be used for both purposes to avoid duplicate fetching.

What to put in robots.txt

This is the common configuration for a business that wants to be found in ChatGPT search and does not want its content used for model training. Adjust it to your own position, which is a commercial decision rather than a technical one.

User-agent: OAI-SearchBot
Allow: / User-agent: GPTBot
Disallow: / User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml

Three checks make this real rather than decorative. Confirm the file is served at the root of the exact hostname people reach, including the www or non www version you redirect to. Confirm your firewall, bot manager or CDN is not blocking those agents at the edge, because an allow in robots.txt means nothing if the request never completes. And confirm the pages you care about return a complete answer in the initial HTML response rather than only after client side rendering.

The full mechanics are in how to write a robots.txt, and the wider decision across every assistant in which AI crawlers to allow.

Reading the documentation well

• Quote figures only from the pricing page and the model catalogue. Those two pages change, and any number taken from an article has a date attached to it that the article rarely shows.
• Prefer the reference over a tutorial for anything about parameters, limits or error codes.
• Check the deprecations page before a migration, and again a week before you ship.
• Treat the Cookbook as patterns. It shows a working approach, and the production concerns such as retries, timeouts and cost control belong to you.
• Verify crawler traffic by IP, not by user agent string alone, since a user agent is easy to copy and the published IP lists are the check that holds.

What the documentation does not decide for you

Allowing OAI-SearchBot makes your pages reachable. Which pages get used in an answer is a separate question, and it is decided by whether your page answers the question somebody actually asked, in language they used, with facts stated plainly enough to be lifted. That is editorial work rather than configuration work, and it is covered in how ChatGPT picks sources. A companion file some sites now publish is covered in how to write an llms.txt.

Where is the OpenAI documentation now?

At developers.openai.com/api/docs. Links to platform.openai.com/docs redirect there, so old bookmarks still resolve.

What is the difference between GPTBot and OAI-SearchBot?

OAI-SearchBot crawls for ChatGPT's search features, and GPTBot crawls content that may be used to train OpenAI's foundation models. The documentation states the two robots.txt settings are independent, so you can allow one and disallow the other.

Should I block GPTBot?

That is a content licensing decision. Publishers whose archive is the product commonly disallow it, while businesses that want to be described accurately in model answers commonly allow it. Allowing OAI-SearchBot is the separate decision that governs appearing in ChatGPT search answers.

Does robots.txt apply to ChatGPT-User?

The documentation says ChatGPT-User fetches are initiated by a user, so robots.txt rules may not apply, and that this agent is not used to determine whether content appears in search. Use OAI-SearchBot in robots.txt to manage the search side.

How long does a robots.txt change take to register?

The documentation notes that it can take around twenty four hours from a robots.txt update for the search systems to adjust. Make the change, then re-check your server logs for those agents over the following week.

How do I confirm the crawler actually reached my pages?

Search your server or CDN logs for the three user agent names, check the response codes they received, and verify the requests against the published IP lists. A run of redirects, timeouts or edge blocks shows up there immediately.

From configuration to being named

Getting the crawler settings right is the entry condition. The question it leads to is what the assistants actually say when a buyer in your category asks, because being reachable and being recommended are two different measurements. Ask the question a customer would ask, keep your own brand name out of the wording so the answer is not steered, and record which companies get named and which sources the answer leans on. AI Knows Us runs that check across ChatGPT, Gemini, Perplexity, Claude and Google's AI Overviews and shows the sources behind each answer.