Screaming Frog is a crawler that runs on my own machine. It starts from one address, follows the links it meets, and files what it read on every page into a table: the title, the description, the response code, the images, the inbound and outbound links. The machine doing the crawling is mine, and that is the whole difference between this and a vendor's dataset.
On the site of a precision machining workshop, the first crawl came back with nothing. The source code was empty: every piece of content arrived afterwards, through JavaScript.
1. The problem it solves for me
A client wants the alt attributes on their images fixed, the text that describes an image to anyone who cannot see it. The request is clear. The question that follows is not: which images, and in what order.
That list cannot be compiled by hand. An automated crawl builds it in one pass, one row per file, with the page that carries it. That is the reason I still open this particular crawler.
2. What I actually pull out of it
On a pre-audit for a wholesaler that sells only to trade customers, the crawl counted 276 images and 247 with no alt attribute. Close to nine in ten. That figure opened the conversation; it decided nothing.
What decides is the same list sorted by page. On an equipment distribution account, I handed it over in three tiers: category and sub-category pages first, then the 30 most-viewed products, then everything else. The full media library was left alone, and rightly so: an alt attribute sitting on a page nobody opens earns nothing.
So what I ask this crawler for fits in one sentence: one precise column, across every address on a site, in a single file. Not a diagnosis.
3. What I never ask it for
To tell me what a search engine sees. The crawl reads the source code as the server sent it, before the browser processes it. JavaScript execution is available, but it is not the starting setting: it has to be chosen, and it only carries from one crawl to the next if it is saved. The vendor puts it that way in its guide to crawling JavaScript websites. On that machining workshop's site the consequence was brutal: an empty page body, no links to follow, a report that made the site look like it did not exist. The site was fine. The crawl was looking in the wrong place.
To set my priorities. It returns a table, and a table does not say what is costing sales. My audit crawler is Sitebulb now, which returns a ranked report; Screaming Frog stayed for extraction. I did not plan that split; it has held.
To run while I get on with something else. In its general user guide the vendor states a minimum of 4 GB of RAM, 8 GB for a crawl under 200,000 addresses, and a fast drive with 16 GB beyond a million. Those are their figures, and they describe what happens accurately: on a large catalogue, the workstation is taken for the length of the crawl.
4. What I run alongside it
A crawl finds a page. It does not say whether the engine kept it. The two lists never quite overlap, and the gap between them is exactly what gets worked on: I have written up what the engine's own source shows, and where its holes are, in what I ask Search Console for.
The rest splits along the same line. DataForSEO says what people are searching for, Ahrefs says who cites whom. Neither one reads the code of a page; the crawler reads nothing else.
On an SEO engagement this crawl is the first move, before any recommendation at all. On a WordPress site it also measures what a page builder left behind, and I have written what that looks like in the technical debt of a builder-based site and in taking over a site an agency left behind.
5. What it costs at my scale
The cost of a desktop crawler does not show up on the invoice. It shows up in two places.
The machine first. The vendor's figures are not decoration: a catalogue running to several hundred thousand addresses holds the workstation for as long as the crawl lasts. Those crawls, I start at the end of the day.
Then the expiry date. The file describes the site as it stood on the day of the crawl. Two releases later, it describes a site that no longer exists. That is what rules out crawling a site once and for all at the start of an engagement: the crawl gets repeated, or it lies.
The setting makes the result
A crawl report does not describe a site. It describes what one machine asked for, with the settings it was given, on the day it was run. Faced with an empty table, the first instinct should not be to conclude that the site is empty. It should be to look at what was asked.
Does your site come back empty from a crawl?
I run the crawl, compare what it finds against what the engine kept, and hand back the list of gaps, sorted by what they cost. It is the first move on my SEO engagements and on my WordPress work.
Request a free audit