Sign inStart free

Best SEO spider software compared: the crawlers, and the limit that sets each tier

Desktop against cloud, the published crawl ceilings, what JavaScript rendering costs you, and the first four reports to read after any crawl finishes.

Published by AI Knows Us · Updated 21 October 2026

A spider is software that follows the links on your site the way a search engine would and records what it finds at every URL: the status code, the title, the canonical, the directives, the depth, the links in and out. Everything else a crawler offers is built on that one table, and the choice between crawlers comes down to two questions. Where does it run, and where does it stop.

The criterion on this page is the second one: does the vendor publish the ceiling at which its crawl stops, in a number you can check before paying? That is the fact that decides whether a tool fits your site, and it is also the fact most feature grids leave out.

Desktop or cloud, which is the real decision

Desktop spiderCloud crawler
Runs onYour own machine, using your bandwidth and memoryThe vendor's infrastructure, on a schedule you set
StrengthFull control, staging and password protected sites, repeat crawls while you fix somethingIt runs on Tuesday whether you remember or not, and it diffs against last Tuesday
WeaknessA large crawl competes with everything else on your laptopReaching a staging environment is somebody's firewall problem
Right forOne person doing the work, and any site under a few thousand URLsA team that reports, and a site where something breaks every release

The published ceilings, which is where this ranking comes from

Screaming Frog SEO Spider is first on this criterion and it is first by a distance, because its own page publishes the whole split. Read on 21 October 2026, that page states the free version crawls up to 500 URLs in a single crawl and needs no sign up or email capture. It then names, item by item, what the annual licence unlocks: the crawl configuration itself, saving and reopening crawls, JavaScript rendering, crawl comparison, near duplicate content, custom robots.txt, structured data validation, spelling and grammar checks, custom source code search, custom extraction, a custom JavaScript crawl with OpenAI and Gemini, and the Analytics, Search Console and PageSpeed integrations, plus accessibility auditing, segmentation and a Looker Studio report. It also states the tool audits for over 300 issue types. The licence is annual and renews.

That level of publication is the thing to look for in any crawler you consider. A buyer can read it once and know whether free covers their whole site, which for a few hundred pages it does.

Second, and on a different footing, is Search Console. It is not a spider you point at a URL, so it does not compete on this criterion at all, and it is still the first thing to open because it reports Google's own crawl rather than an imitation of it. Use both: the spider finds what is wrong, Search Console says whether Google noticed.

Third are the cloud crawlers inside the suites. The useful question with those is never the issue list, which is broadly the same everywhere, but the crawl allowance on your tier and whether JavaScript rendering is inside it, because rendering is the setting that multiplies crawl cost everywhere it appears.

The first four reports to read when a crawl finishes

In this order, because a finding near the top makes the ones below it irrelevant. Status codes first: fix the 404s and collapse the redirect chains. Then indexability: noindex tags, canonicals pointing somewhere you did not intend, and anything disallowed in robots.txt. Then duplicate and missing titles, which is almost always one template rather than fifty pages. Then depth and orphans: anything more than three clicks from the homepage that you expect to sell something is a navigation problem rather than an SEO problem.

A crawler reads your server, and one answer is not on your server

Everything a spider can tell you comes from HTML you serve. When a buyer asks an assistant who to buy from, the answer is composed for that asker from sources the assistant chooses, so it exists nowhere in your site for a crawler to fetch. Screaming Frog is honest about the boundary: the AI feature its page names is a custom JavaScript crawl with OpenAI and Gemini, which is a model helping inside your crawl, not a measurement of what an assistant says about you.

AI Knows Us measures that second thing and nothing else on this page: your chosen purchase questions, put to ChatGPT, Gemini, Perplexity, Claude and Google AI Overviews, with every named business logged each time. The billing unit is on our pricing page for anybody to read, and it is one question on one app on one occasion, with Claude counted at three because Claude costs more to run, and a Visibility API included on the larger plans. We do not sell a crawler and never have, so if you need a spider the table above is the answer. Ahrefs reads more of the assistant surface than we do, including AI Mode and Copilot, and publishes its own per engine cost of 8 checks per update for Claude on its Brand Radar page.

If you came here because somebody told you to crawl your own site to find out why assistants ignore it, the two jobs are worth separating before you spend an afternoon on the wrong one: alternatives to Screaming Frog for AI visibility, and what a crawl of your own server does. We counted durable inbound links across 1,655 tracked pages here on 21 October 2026: 696 had none, and that page was one of them.

What people ask about SEO spiders and crawl limits

Is the free crawler enough for my site?

Count your URLs. Under 500 in a single crawl, the free tier covers the whole thing, and that number is published rather than inferred. Over it, you are choosing between a licence and crawling in sections.

Do I need JavaScript rendering?

Only if your main content or your links appear after scripts run. Check it once with Search Console URL Inspection, which shows the rendered HTML Google holds, then you know for certain instead of paying for rendering as insurance.

How often should I crawl?

Monthly for a stable site, and after every release, migration, theme change or platform upgrade. Those events are when crawlers earn their keep, because they find a template level mistake while it is still cheap.

Will crawling my own site slow it down?

It can, which is why every serious crawler lets you limit speed and threads. Crawl a production site slowly outside business hours, or crawl staging, and the question stops being a question.

Can a spider find pages nothing links to?

Not by following links, by definition. Feed it your sitemap as well as your homepage, and compare the two lists: the pages in the sitemap that the link crawl never reached are your orphans, and that comparison is the most valuable five minutes in any audit.

Read next

SEO tools for a two to four person in house team in Singapore, chosen on the four units that set the billVendor pages read on 28 October 2026, the seat arithmetic at three people, and why the crawl ceiling and the site count decide this rather than the brand.Googlebot: what the crawler is, how it behaves on your site, and how to verify a request really came from GoogleGooglebot is two crawlers under one robots.txt token. Here is what Google documents about its limits and behaviour, how to control it, how to verify it, and what the other crawlers in your logs are doing.Simon-Kucher pricing strategy, what the firm publishes about its own method and what a Singapore buyer should ask before briefing itThe firm publishes its approach, its scale and four named service lines, and publishes no fee, so the useful work before any call is reading what it claims and deciding which decision you are buying.International pricing strategies for a Singapore company, and the corridor that stops one market undercutting anotherA converted home price is the one answer that is almost always wrong, so set a price per market and keep the gaps narrower than the cost of moving a unit between them.Premium pricing strategy for a Singapore business, and the fence that makes the higher number holdCharging permanently more than the credible alternatives works when the buyer can name what the extra money buys, and the published Singapore writing ladder shows exactly how that is built.Red Antler publishes no price, so here is what its own site does publish and how to cost the same scope from SingaporeRead on 29 October 2026, three pages of redantler.com carry a complete deliverable list, three business stages and not one figure about what any of it costs.

Or browse the whole archive.

See what AI says about you.

The first scan is free, it needs no account, and it asks the questions your buyers ask.

Free. No card. We ask 5 real buyer questions on 2 AI apps.

Already sure you want this? Start free. No card needed, and the free plan stays free. Pay only when you want the work done. Or talk to a person if you have a question first. Or read how we fix what it finds.