Screaming Frog SEO Spider: the setup, the first crawl, and the reports to read in order

Screaming Frog SEO Spider is a desktop crawler you install on your own machine. You give it a starting URL, it requests every page, image, script and link it can reach, and it hands you a spreadsheet of what the site actually returns: status codes, titles, meta descriptions, canonicals, redirects, headings, word counts and the internal links pointing at each URL. The free version crawls up to 500 URLs. A licence removes that limit and adds the features most audits need, and it is sold per user as an annual licence at GBP 199 for one to four licences, with lower per licence rates at five, ten and twenty licences, as published on the Screaming Frog pricing page on 9 October 2026.

This page covers how to set the crawler up, the first crawl, and the reports to read in the order that finds real problems. For the wider list of checks an audit should cover whatever tool you use, see the technical SEO checklist.

What the free version covers, and what the licence adds

The split matters, because plenty of small sites never need the paid version. As published on the vendor pricing page on 9 October 2026:

Capability	Free version	Licence
Crawl limit	500 URLs	Set by the memory and storage you allocate
Broken links, errors and redirects	Yes	Yes
Titles, meta data, meta robots, hreflang	Yes	Yes
Exact duplicate pages	Yes	Yes
XML sitemap generation and visualisations	Yes	Yes
Saving and reopening a crawl	No	Yes
Crawl configuration and custom robots.txt	No	Yes
JavaScript rendering	No	Yes
Crawl comparison and near duplicate content	No	Yes
Structured data validation, spelling and grammar	No	Yes
Custom extraction, custom source code search, custom JavaScript	No	Yes
Search Console, Analytics, PageSpeed Insights and link metrics integrations	No	Yes
Scheduling, segmentation, Looker Studio crawl report, accessibility auditing	No	Yes

Two of those lines decide most buying questions. Not being able to save a crawl means every session starts again, which is fine for a 40 page site and painful for a 4,000 page one. Not having the Search Console integration means you are looking at what the site returns without the impressions and clicks that tell you which of those URLs anyone reaches.

Five settings to change before your first crawl

• Respect your own staging rules. If you are crawling a protected environment, set the authentication before you start rather than collecting 401s for an hour.
• Limit the crawl depth or the total URLs on a first pass. A capped pass of a few thousand URLs tells you the shape of the site and whether your configuration is right, before you commit to the whole thing.
• Decide whether to crawl subdomains. Off by default. Turn it on when the blog, help centre or shop sit on their own subdomain and you want them in the same picture.
• Exclude the infinite spaces. Faceted filters, calendars, search result URLs and session parameters generate a near endless supply of URLs. Add exclusion patterns now, or spend the crawl on them.
• Set the speed deliberately. The default rate suits most hosting. On a small shared server, lower it. On a large site behind a CDN, raise it and watch the response time column rather than guessing.

The first crawl, in order

Start with the HTTPS homepage, with the final hostname. Crawling the version that redirects wastes a hop on every URL and makes the report harder to read.

While it runs, watch the status code distribution in the right hand panel. A rising count of 404s or 5xx tells you to stop and look now rather than at the end. When the crawl finishes, work through the reports in this order, because each one changes what the next one means.

• Response codes. Filter to 5xx first, then 4xx, then 3xx. Server errors are the only finding here that can be costing you pages today. For each 404, use the Inlinks panel at the bottom of the window to see which pages link to it, which is the fix: change the link, not just the destination.
• Redirect chains. Run the redirect chains report. Any chain of three or more hops is worth collapsing to a single redirect, and a redirect that ends in a 404 is a link you are paying for and getting nothing from.
• Canonicals. Filter for canonicalised URLs and for pages whose canonical points somewhere unexpected. This is where a template bug shows up as thousands of pages pointing at the homepage.
• Indexability. The Directives tab shows noindex and nofollow values. Compare the noindex set against what you intended to exclude. Pages carrying noindex by accident are the single most common serious finding in a first audit.
• Titles and meta descriptions. Sort by duplicates rather than by length. Two hundred identical titles is a template problem worth one fix; a long title is worth a rewrite only if the page earns traffic.
• H1s and page depth. Pages more than four clicks from the homepage are usually the ones nobody links to. Cross reference with the internal linking audit.
• Images. Filter for missing alt text and for images over a sensible file size. The second list usually explains a slow template.

Connect the APIs, then crawl again

With a licence, connect Search Console, Google Analytics and PageSpeed Insights before you crawl, and every row arrives with its clicks, impressions, sessions and field performance data attached. That turns a list of technical findings into a prioritised list, because you can sort the 404s and the slow templates by the traffic they used to get or still get.

It also produces the orphan URL report: URLs that Search Console or Analytics know about but that no internal link reaches. Those are pages you are publishing and then hiding.

JavaScript rendering, and when to turn it on

By default the crawler reads the HTML the server returns. If your pages build their content, links or titles in the browser, switch the rendering mode to JavaScript so the crawler executes the page the way a browser does. Compare the rendered HTML against the raw HTML in the view source panel: anything present only after rendering is content that depends on your scripts running.

Rendering is several times slower, so use it for a sample of templates rather than the whole site, then crawl the rest in plain HTML mode.

Large sites: memory and storage

The vendor states plainly that the maximum number of URLs you can crawl depends on the memory and storage you allocate. The practical move for anything above roughly half a million URLs is to switch from memory storage to database storage in the configuration, point it at an SSD, and raise the allocated RAM. Database mode also lets you reopen a finished crawl instead of running it again.

For very large sites, crawl by section instead of all at once. One crawl per template family, with its own exclusions, gives you findings you can act on and avoids a single run you never finish reading. Crawl budget work follows the same logic.

The Screaming Frog SEO Spider questions worth an answer

Is Screaming Frog free?

There is a free version with a 500 URL crawl limit and the core reporting. The licence removes the limit and adds saving crawls, JavaScript rendering, the Google API integrations, custom extraction and the rest. The vendor publishes the annual per licence price on its pricing page.

How long should a crawl take?

Plain HTML crawling at the default rate moves through a few thousand URLs in minutes on healthy hosting. Rendering mode is much slower. If a crawl slows down, check the response time column before you raise the speed, because the site may be the limit rather than the tool.

Does crawling my own site cause any harm?

A crawl is ordinary HTTP traffic. Keep the thread and rate settings moderate on small hosting, crawl outside your busiest hours, and exclude the endpoints that trigger heavy database work, such as internal search.

What is the difference between this and a cloud crawler?

A desktop crawler uses your machine and your connection, which makes it fast to start and easy to point at a protected environment. A cloud crawler runs on a schedule without your laptop being open and keeps a history for everyone on the team. Many teams run both: the desktop tool for investigation, a scheduled crawl for monitoring.

Which single report should I read first?

Directives, filtered to noindex. It takes one minute and it is the check that most often finds a page or a whole section being excluded from search by accident.

Can it replace Search Console?

No. A crawler tells you what your site returns. Search Console tells you what Google did with it. The audit that holds up uses both, which is why the integration is worth setting up.

The crawl you cannot run yourself

Screaming Frog crawls your site from your machine and tells you what a crawler finds. It cannot crawl an assistant's view of you. That view is what the assistant has retained and retrieved, and it may include pages you consider minor and omit ones you consider central.

The inspection is simple. Ask each assistant what your site offers, then compare with your crawl. Gaps show which pages are doing the work. AI Knows Us gathers the answers and the cited URLs so they can be compared with your export.