toolkit

Scenario · The bigger jobs

The crawl found four hundred problems

A full crawl comes back with hundreds of findings and somebody has to decide what matters.

A crawl that comes back with hundreds of findings is not a to-do list — most of those findings are the site working exactly as intended, and triaging all of them by hand takes a day and produces a different answer depending on who's doing it. This sorts every finding into fix, safely ignore, or needs a person to decide, each with a stated reason, so the actual work is a short, real list instead of a wall of noise nobody has the stomach to read.

What to ask for

Site crawl (Screaming Frog)Full-site crawl audit powered by Screaming Frog — the heavyweight, whole-site counterpart to the per-page SEO skills.Crawl-finding dispositionCrawl-finding disposition — a crawl tells you what is true; this decides what is worth doing.Sitemap & robotsRead-only sitemap & robots map — fetches a site's robots.txt and every sitemap it declares (or probes the common locations), follows sitemap indexes down to their children, collects every URL, and reports what is declared, what robots.txt blocks, and what is oversized, cross-host or insecure, plus a crawl-over-crawl diff.On-page SEOOn-page SEO scan — reads a page's delivered HTML and checks the head, meta and heading signals search engines use: title presence and length, meta description, rel=canonical, meta robots (noindex), Open Graph tags, viewport, html lang, and exactly one H1, plus a structured-data summary.

See it work

A real run of Crawl-finding disposition:

PASS  1 fix · 0 needs-human · 3 dismiss
  [info] 3 of 4 finding(s) (75%) need no work — each with a stated reason. That is the half of crawl triage that was being done by hand

  report: ./out/crawl-finding-disposition-crawl-finding-disposition.html
  csv:    ./out/disposition.csv

The captured report, exactly as a run hands it to a client —open the full report ↗

A crawl’s output is not a to-do list, which is the mistake that makes crawls useless. Four hundred findings triaged by hand takes a day and produces inconsistent decisions; most of them are the site working as designed.

  1. Get the URL set right first. The sitemap-and-robots read establishes what the site declares. A crawl scoped differently from the declared site produces findings about URLs nobody intended to publish, which is noise that looks like signal.
  2. Run the crawl. The deep crawl is the data-gathering step and it is the cheap part. Resist reading it directly — a raw crawl export is where triage goes to die.
  3. Disposition every finding before touching one. The crawl-disposition pass classifies each as fix, dismiss, or needs-human, with the reason stated. The classification turns on what kind of URL it is: noindex on a paginated archive, a tag page or a thank-you page is correct behaviour; the same noindex on a service page is a page removed from Google. Duplicate headings and crawler spelling flags are dismissed with a reason, because they are hand-dismissed nearly every time.
  4. Read the needs-human queue, and only that. This is the actual work product — the findings where the right answer depends on what the page is for. It should be a short list; if it is not, the disposition rules need tuning rather than the site.
  5. Sanity-check the fix queue against the pages that matter. Grade on-page SEO for the practice’s top service pages directly. A crawl finding on a high-value page outranks fifty on archive pages, and crawl output is not ordered by commercial importance.
  6. [manual] Work the queue, routing each item to its own procedure. Broken links, redirects, missing titles, thin content — each has a scenario. This one hands them over.
  7. Re-crawl and diff, not re-crawl and re-read. The value of the second crawl is what changed.

Dismissed-with-a-reason is the deliverable

A dismissed finding with a stated reason can be reviewed and can be reversed. A finding that was simply never looked at is indistinguishable from one nobody noticed. The disposition pass exists because that distinction is the difference between a crawl that gets run again and one that does not.

What this does not cover

Fixing anything. This ends with a queue, not a repaired site — each fix then follows its own procedure. It also cannot tell you a finding matters commercially: a noindex on a page nobody should index is the site working correctly, and a noindex on the practice's highest-value service page is a catastrophe, and the crawler reports both identically.

← All scenarios