GitHub as a search channel, what its robots file lets a crawler read, what it blocks and why the README is the page that gets found

For a company that publishes code or developer tooling, GitHub is a search channel rather than a code host, and the page that does the work is the repository README. GitHub own robots file, fetched on 22 October 2026, permits crawling of the repository page while disallowing the file browser, the raw file view, the commit history and the blame view. So the content inside your files is not what gets read. The README on the repository front page is, which means it should be written as a page for a human stranger rather than as installation notes for somebody who already decided.

Harvested against the India geo target on its own on 22 October 2026, github seo draws 70 searches a month in India, and against the United Kingdom target on its own on the same day it draws 20 a month in the United Kingdom. Two countries, read separately.

What the robots file says, measured

We fetched https://github.com/robots.txt on 22 October 2026. HTTP 200, 14,540 bytes, 13 user agent lines. Two things in it are worth knowing before you plan anything.

First, one rule group names five AI crawler identities together: GPTBot, OAI-SearchBot, ClaudeBot, anthropic-ai and PerplexityBot, and gives that group Crawl-delay: 1 followed by the same allow and disallow list the general group gets. GitHub is neither blocking them nor treating them specially. It is pacing them.

Second, the disallow list tells you where the crawlable surface ends. These paths are disallowed in that group: /gist/, /search and /search/advanced, /*/raw/, /*/tree/, /*/blame/, /*/archive/, /*/*/commits/, the tarball and zipball paths, anything under /.git/, any .atom feed and /copilot/. The repository root, where the README renders, is not in that list. The file does also publish a sitemap, at a marketing sitemap URL, and a comment inviting anybody who wants to crawl GitHub to contact support.

A robots file is a published instruction rather than an access control, so this describes GitHub stated policy on the date we read it.

Why that matters, with a number beside it

In our own second audit, 630 questions per market across three markets producing 1,260 readings on 16 October 2026, github.com appeared in 243 of those 1,260 readings, which placed it sixth among the ten domains that won that audit. It was classified as social or community rather than as a vendor site. Two engines, ChatGPT and Gemini, not five, and that distinction holds in every sentence.

So a domain whose file browser is closed to crawlers was nonetheless one of the most frequently cited sources in a run of a thousand two hundred and sixty answers about marketing and search. The surface doing that work is the readable one: repository front pages, their READMEs, and the docs site.

What a README has to do, given the above

• Answer what this is in the first two lines. Not what it installs. What problem it solves and for whom. This is the text most likely to be read in isolation.
• Put the comparison in the README. How this differs from the obvious alternative, named neutrally. A stranger arriving from a search result is choosing, not installing.
• Include the worked example in full. Not a link to an examples folder under /tree/, which is disallowed. If the example matters, it lives in the README.
• Carry a date and a version. A README with no date reads as abandoned, and that judgement is made in seconds.
• Link out to your own documentation site. That is where depth belongs, and it is a domain you control.

GitHub Pages, and the decision it forces

GitHub Pages publishes a site from a repository, which makes it the cheapest way to get a documentation site online. The decision it forces is the domain. A docs site on a GitHub subdomain accumulates its reputation there. The same site on your own domain accumulates it for you. For anything commercial, map a custom domain from the start, because moving it later means redirecting URLs that other people have linked.

The wider lesson about restating somebody else rules

Every fact on this page has a date against it for a reason. Crawler lists and robots files change without announcement, and a restated rule goes stale into a false claim. The practical habit is to re-read the file rather than the article about the file. The documentation Google publishes itself, and what each document settles makes the same argument for search guidance, and it is the same discipline applied to a different publisher.

What to do this week if you publish code

1. Fetch your own repository URL and your own docs site and confirm the content a stranger needs is in the response rather than assembled afterwards.
2. Rewrite the first two lines of your main README for somebody who has never heard of the project.
3. Move any decisive example out of a linked folder and into the README itself.
4. Map a custom domain to your Pages site if you have not.
5. Put a last updated date on the README and the docs index.

Questions about GitHub and search visibility

Can a repository rank for a general search

For a specific tool name or a precise technical phrase, routinely. For a broad commercial phrase, your own site is the better bet, because the repository page is competing as one page rather than as a site with internal structure.

Does starring or forking affect how findable a repository is

Those are signals inside GitHub own ranking and discovery, which is a separate system from a search engine index. Treat them as social proof a human reader notices on arrival rather than as a search lever.

Should documentation live on GitHub or on our own site

Your own site, with the README as the front door that points at it. The reasoning is ownership rather than quality: a domain you control is one you can redirect, measure and keep.

Are the AI crawlers blocked from GitHub

Not in the file we read on 22 October 2026. The group naming GPTBot, OAI-SearchBot, ClaudeBot, anthropic-ai and PerplexityBot carries a one second crawl delay and the same path rules as the general group, which means the repository page is permitted and the file browser, raw view and history are not.

Is a gist a useful place to publish anything findable

No. /gist/ is in the disallowed list in that file, so treat a gist as a way to share a snippet with somebody who already has the link.

What an assistant says about your project when nobody links it

The reason the README matters more than it used to is that a growing share of the question gets answered without anybody visiting a page. Somebody asks an assistant which library or which tool to use for a job, and the answer names two or three, assembled from the sources it could read. Our own figures above show how much of that reading passes through this one domain. At AI Knows Us we measure, dated, whether the five assistants name a project or a company when the buying question is asked, which for anybody publishing developer tooling is the difference between being on GitHub and being chosen from it.