Check a Singapore marketing supplier's own robots file before you hire it, and what twenty nine of them say

Open a supplier's domain, add /robots.txt to it, and read what comes back. That is a one minute check, it costs nothing, and for a supplier selling AI search visibility it tells you whether the firm has applied its own advice to its own website. We ran that check on 27 October 2026 across twenty nine Singapore marketing suppliers and directories, all of them hosts that ChatGPT or Gemini had cited when a Singapore buyer asked a buying question. All twenty nine served a file with an HTTP 200. Five had written an explicit rule for the crawler that OpenAI says governs whether a site can appear in ChatGPT's search answers. Twenty four had not.

The four OpenAI agents, and the only one that decides search inclusion

This check is worth running because the agents do different jobs, and OpenAI documents which is which. Read on developers.openai.com on 27 October 2026.

• OAI-SearchBot surfaces websites in ChatGPT's search features. Sites that opt out are excluded from ChatGPT search answers, although they can still appear as navigational links.
• GPTBot crawls content that may be used to train OpenAI's foundation models. Disallowing it signals that content should not be used for that training.
• ChatGPT-User handles user initiated actions, such as visiting a page because somebody asked a question. The documentation states it is not used for automatic crawling and does not determine whether content appears in search.
• OAI-AdsBot checks the safety of landing pages submitted as ads on ChatGPT and may use their content to judge when an ad is relevant. It visits only submitted ad pages.

The documentation adds two operational facts worth writing into any brief. Each setting works independently of the others, and robots.txt changes take about 24 hours to take effect. Perplexity publishes the same shape of guidance for its own agents, which we cover separately.

What twenty nine Singapore files actually contained

Measured on 27 October 2026. The sample is the Singapore supplier and directory hosts cited by ChatGPT or Gemini across the Singapore agency, pricing, paid media, vertical and generative engine batches of our 26 October audit. All twenty nine returned a file.

The five that name OAI-SearchBot and allow it are aistudio.com.sg, techdirectory.sg, www.firstpagedigital.sg, www.digitalmarketingagency.sg and outrankco.sg. One more, mediaplus.com.sg, allows GPTBot and ClaudeBot but names no OpenAI search agent, which is the pattern of a site that has thought about model training and not about search inclusion. Those two decisions are separate in OpenAI's own documentation, and this is what the difference looks like on a live Singapore site.

What an absent rule does and does not mean

This is the part a sales conversation will get wrong in both directions, so it is worth stating precisely. A file that does not name OAI-SearchBot is not blocking it. Where no disallow rule applies, the crawler is permitted. Naming the agent is a declaration, not a requirement. So the twenty four Singapore files that say nothing are not shutting the engines out. What they show is that no decision has been recorded, and for a supplier that sells this service the absence is information about the supplier rather than about the technology.

One of the five is admirably direct about the limits of the file. The robots file on www.firstpagedigital.sg carries its own comment, read on 27 October 2026, stating that robots.txt is advisory and does not guarantee crawler behavior, training exclusion, AI rankings, citations, or discovery. That is an accurate sentence and a competitor wrote it. A buyer should want the same honesty from whoever they hire.

How to run the check on a supplier, in five minutes

1. Read their file. Visit the supplier's domain with /robots.txt appended. Note whether OAI-SearchBot, PerplexityBot and ClaudeBot appear at all, and whether the rule beside each is an allow or a disallow.
2. Look for a sitemap line. A file with no sitemap reference on a site selling discovery work is a small signal and a fair question to ask.
3. Check for an llms file. One of the twenty nine, techdirectory.sg, publishes an LLM-Content line pointing at its own llms.txt, and that file returns a 200 and describes the site's sections in plain language. Whether you want one is a separate decision, and a supplier that has implemented one on its own site can show you the result rather than describe it.
4. Ask about the 24 hour clock. A supplier who knows that a robots change takes about a day to take effect is working from the documentation. A supplier promising a same day change in what an assistant says is not.
5. Ask what they will not claim. The honest answer is that access is a permission, not a placement. Access makes you readable. Being named is decided by what the engines read, and in Singapore that is mostly third party pages.

Now run the same check on your own site

The same five minutes applied to your own domain answers the question most Singapore buyers actually arrive with, which is how to get an assistant to recommend their company when somebody asks for a supplier. Three steps in order.

• Permit the search agent. Make sure nothing in your file or at your edge blocks OAI-SearchBot or PerplexityBot. If your site sits behind a bot management product, the rule that matters may be at the edge rather than in the file, so check both. Perplexity's documentation also recommends permitting requests from its published IP ranges, which is an edge setting rather than a robots line.
• Decide training separately. GPTBot and Google-Extended govern model training, not search inclusion. Blocking them is a legitimate commercial decision and it does not remove you from ChatGPT's search answers.
• Then do the work that gets you named. In our 26 October 2026 audit of 64 blind pipelines across 32 batches with Singapore locked, the engines cited 321 distinct hosts, 94 of them on Singapore domains, and no AI visibility product was named for a Singapore buyer. Access is what makes you eligible. Presence on the hosts the engines read is what gets you into the answer.

If you want the reading rather than the theory, AI Knows Us shows which sources an answer was built from for a given question, and the scan at aiknowsus.com runs without an account, so you can see your own source list before deciding whether any of this needs a supplier. Plainly, though, the robots check on this page needs no vendor at all, ours included, and a buyer who only does that check has still done the most cost effective thing available this week. For the monitoring side of the same question, our comparison of tools that track ChatGPT mentions sets out what each vendor can actually see.

Questions Singapore buyers ask about crawler access and ChatGPT recommendations

If I allow OAI-SearchBot, will ChatGPT recommend my company

It makes you eligible to be surfaced in ChatGPT's search features, which is what OpenAI's documentation says the agent governs. Whether you are named in an answer depends on what the engine reads about your category, and in Singapore that source set is dominated by third party pages rather than brand sites.

Should a Singapore business block GPTBot

That is a content licensing decision rather than a visibility one. OpenAI's documentation separates training from search, so you can allow the search agent and disallow the training agent, and several of the twenty nine files we read do exactly that in one direction or the other.

Does a supplier with no AI rules in its robots file know what it is doing

Not on its own, no. Twenty four of twenty nine Singapore files we read name no AI agent, so the absence is the market norm rather than a disqualification. Use it as one question in a conversation, and weigh the answer rather than the file.

What is the Content-Signal line that two of them publish

It is a declaration beside the user agent rule stating how the site is willing to have its content used, for example for training, search and AI input. Two of the twenty nine publish one. It is a statement of intent rather than an enforcement mechanism, and it is worth knowing about because it will come up in proposals.

How often should this be checked

Quarterly for your own site, and again after any platform migration or security product change, because an edge rule added for bot protection can block a search agent without anybody editing the robots file.

The file is the cheapest thing on your shortlist and the only one you can read without asking

Everything else a Singapore supplier tells you about this work arrives through a proposal. The robots file arrives through a browser, in five minutes, with no meeting and no account. Twenty nine Singapore suppliers served theirs on request on 27 October 2026 and five of them had written down exactly what they allow. Read the file of the next firm that pitches you this service before the call rather than after it, and the first question of the meeting will be a better one.