Screaming Frog SEO Spider: Run a Reproducible Site Crawl

Screaming Frog SEO Spider is a desktop crawler for Windows, macOS and Linux. It is well suited to analyst-led audits that need local control, detailed URL exports and custom configuration. Save the crawl settings with your evidence so another person can reproduce the result.

What Is Screaming Frog?

The official product page documents checks and exports for links, response codes, directives, canonicals, page titles, metadata, duplicates, hreflang, structured data and other crawlable elements. The SEO Spider is separate from Screaming Frog's Log File Analyser.

Free vs Paid Version

The free version crawls up to 500 URLs in one crawl. Screaming Frog states that saving crawls, JavaScript rendering, custom extraction, API integrations, scheduling and other configuration features require a paid licence. The official page listed £199 per licence per year on 11 September 2026; confirm current pricing before purchase.

Run Your First Crawl

1. Confirm written permission, intended host and whether subdomains or staging are in scope.
2. Set a considerate crawl speed and exclusions for traps such as internal search, calendars or infinite parameter combinations.
3. Choose Spider mode for link discovery or List mode for a controlled URL set.
4. Start with HTML crawling. Switch to JavaScript rendering when important content or links depend on client-side rendering.
5. Compare the crawl with XML sitemaps, analytics or Search Console exports to find URLs that links alone did not reveal.

Read the Main Tabs

• Response Codes: filter by status, then export the inlinks that led to a broken or redirected URL.
• URL, Page Titles, Meta Description and H1: group findings by template before editing.
• Canonicals and Directives: compare each signal with the URL's intended index state.
• Images: identify large files and missing alt attributes, then inspect purpose and context manually.
• Internal: build a URL-level evidence set across the collected fields.

Use Custom Extraction

The paid version can extract data with XPath, CSS selectors or regular expressions. Use it for repeatable checks such as product identifiers, author fields, breadcrumbs or component markers. Validate the selector on several templates, include an expected value, and retain the extraction configuration with the audit.

Integrate GA and GSC Carefully

The paid product documents Google Analytics, Search Console and PageSpeed Insights integrations. Align dates and dimensions before treating a row as comparable. Analytics activity does not prove indexability, and a Search Console impression does not prove that the current page implementation is correct.

Advanced Configuration

Use rendering, authentication, custom robots rules, alternate user agents and API connections only when the scope requires them. Record each non-default setting. For large crawls, choose memory or database storage deliberately and monitor server response.

Best-Fit Tasks

Screaming Frog fits migration validation, broken-link tracing, metadata and directive inventories, rendered-versus-raw comparisons and custom source extraction. A cloud platform may fit better when the main need is shared dashboards or hosted monitoring. Compare deployment and evidence requirements in the audit tool comparison.

Capabilities and price checked against Screaming Frog's official product pages on 11 September 2026.