Web crawler is also known as a spider, a bot or a robot: the names and what each means

A web crawler is also known as a spider, a spiderbot or simply a bot. Older writing adds ant and automatic indexer, and some texts say web robot. All of these names describe the same thing: a program that visits web pages by following links, reads what it finds, and passes it on to be stored or analysed. When a search engine does this to build its index, the crawler is called a search engine spider. Googlebot, which is Google's crawler, is the best known example.

The names matter because they show up in different places. "Spider" is common in old SEO writing and in tool names. "Bot" is what you see in server logs and in robots.txt files. "Crawler" is the word in most documentation. This page lists the names, separates the kinds of crawler that share them, and shows what each kind means for a site owner.

The names, and where each one is used

Name	Where you meet it
Crawler	Search engine documentation, SEO tools, technical guides
Spider	Older SEO articles and tool names such as the Screaming Frog SEO Spider
Spiderbot	Encyclopaedia entries and academic writing
Bot	Server logs, robots.txt files, security tools
Robot, web robot	The robots.txt standard, the robots meta tag
Ant, automatic indexer	Early research papers on search engines

Wikipedia's entry opens by calling a web crawler an Internet bot that systematically browses the World Wide Web, typically operated by search engines for web indexing, and notes that it is sometimes called a spider or spiderbot and often shortened to crawler. Its section on nomenclature adds the older names, ant and automatic indexer.

How a crawler works

1. It starts with a list of addresses, such as pages it already knows and sitemaps site owners have submitted.
2. It requests a page and reads the response.
3. It collects the links on that page and adds new addresses to its list.
4. It passes the content to another system, which decides what to store, how to understand it and where it should rank.
5. It returns later to check for changes.

Crawling and indexing are two separate steps. A crawler fetching your page does not mean the page will appear in results. Google's own Googlebot documentation, covered in our page on Googlebot, describes the crawler as the part that discovers and fetches pages.

Four kinds of crawler that share the name

Search engine crawlers

These build the indexes behind search results. Googlebot is Google's. Bingbot is Microsoft's. They identify themselves, follow the rules in robots.txt and visit often.

Archive crawlers

These save snapshots of pages so they can be browsed later as they were. Wikipedia describes web archiving as one of the purposes of crawlers.

Scrapers

These copy data from pages for another use, such as price comparison or data sets. Wikipedia lists web scraping among the uses of crawlers. Some identify themselves; many do not. Malicious ones often omit identifying information or disguise themselves as browsers.

AI crawlers

These collect pages for AI products. Some gather training data, and some fetch a page live when a person asks an assistant a question. OpenAI's GPTBot is one example. Each has its own name in the user agent field and its own robots.txt token, so you can allow one and block another.

How to tell which crawler is on your site

A web crawler identifies itself to a server using the User-agent field of the request, and administrators read these fields in server logs to see which crawlers visited and how often. This is how the names in the table above show up in practice: a log line contains a string such as Googlebot, and your robots.txt file can address it by name.

Because a crawler can claim any name, check one that matters before you trust it. Google documents a way to verify that a request really came from Googlebot, and the steps are in the Googlebot page linked above.

What a site owner does about crawlers

• Let the crawlers you want in. A page that is blocked cannot be read, and a page that cannot be read cannot be quoted.
• Write a robots.txt file on purpose. The rules go by crawler name. Examples you can copy are in how to write a robots.txt.
• Link every page you want found. Crawlers follow links, so a page nothing links to is found slowly or not at all.
• Use your own crawler. A tool that crawls your site the way a search engine does shows you broken links, redirects and missing titles. The first crawl is described in the Screaming Frog guide.
• Decide about AI crawlers separately. Search and AI crawlers are different programs. A dated test of which to allow is in which AI crawlers to allow.

Common mix ups about the name

Crawler and scraper. A crawler discovers pages by following links. A scraper extracts specific data from pages. One program can do both, but the intent is different, and the rules for each differ.

Crawler and browser. A browser shows a page to a person. A crawler reads it for a machine, and may not run all the scripts a browser would.

Crawler and index. The crawler fetches. The index is the store of what was kept. A site can be crawled and still not be indexed.

Questions about what a web crawler is called

Is a web spider the same as a web crawler?

Yes. Spider is the older term, and it comes from the way the program moves across the web by following links.

What is a bot in this context?

Bot is the shortest name for any automated program. A crawler is a bot that follows links to find pages.

Is Googlebot a crawler or a spider?

Both words describe it. Google's documentation calls it a crawler and describes two types, a smartphone one and a desktop one.

Do all crawlers obey robots.txt?

The major search engines and AI companies publish their crawler names and follow the rules. Scrapers that disguise themselves do not, so use access controls at the server for anything private.

Can I see which crawlers visit my site?

Yes. Your server logs record each request with its user agent, and many hosts show a summary.

The crawlers behind an assistant's answer

When someone asks ChatGPT or Perplexity about your business, the answer can depend on whether an AI crawler could read your pages at all, and on which of those pages it found first. That makes the crawler list in your logs a short answer to a bigger question: what does an assistant actually say about us? AI Knows Us (aiknowsus.com) asks that question across the assistants and shows the pages each answer was built from, which tells you whether a blocked crawler is the reason you are missing.