toolkit

AI search visibility · Playbook

AI search visibility — the collection → report pipeline

How to produce an AI Search Visibility report end to end. Collection is engine- and org-agnostic; chatot.mjs owns everything from the saved data onward. Nothing here assumes a particular AI vendor, data warehouse, or company — where a team actually collects and stores this data is a local detail kept out of the skill.

Output per client: report.html + report.pdf + report.md (+ report.txt, chatot.json, faq-worksheet.md), all built from the saved data files.

The four questions the report answers

  1. Which AI searches recommend the business (§02)
  2. Which don’t (§03)
  3. Why — the reasoning behind the results (§04)
  4. What we’d build / are doing about it (§06 + Next steps)

Plus: AI chat answer mockups (§02b), Bing/Copilot citations (§05), technical readiness (§07).

Phase 1 — Confirm the business and build the query set

Verify the domain first. A wrong domain silently nulls all citation detection — the engine answer may name the business but the citation check looks for the wrong host and reads 0. Check against the live site before trusting any citation number.

Build ~25–45 queries in how-people-actually-ask-AI phrasing, mixed by category:

  • brand (≈3): the business name, “ reviews”, “ ”.
  • local_commercial / discovery (≈20–30): “best in ” and the natural variants, weighted to the strongest geo, covering every revenue service line. Don’t inherit another client’s query pool — prune services the business doesn’t offer.
  • informational + comparison (≈6–8): “what is ”, “ vs ”. These score ~0 for everyone; they document the gap and seed the FAQ worksheet (phase 5). Keep them as gap-proof samples, not padding.

Phase 2 — Run the queries on each engine

chatot run / chatot collect do this for you: each query goes to each engine in scope (ChatGPT via OpenAI web search, Gemini via search grounding) with live web search, one run per query per engine (n=1), and the raw response is scored directly. Set each teammate’s OPENAI_API_KEY / GEMINI_API_KEY in the env or a gitignored .env first; chatot collect --dry-run <dir> shows the plan and key status without making calls. See references/deep/collection-routes.md for the API details and alternate routes.

Phase 3 — Score and build the data files

Re-derive scoring straight from each raw response — don’t trust stored flags:

  • named / showing — is the business named in the answer?
  • cited — does the answer link to the business’s own site or listing?
  • listed / ranked — is it placed in a numbered/bulleted shortlist (with a rank)?

Also capture the competitors named in each discovery answer. Then aggregate to the two files chatot reads first:

  • query-analysis.json — coverage (the 3-check counts per category per engine), tiers, wins, gaps, by_query.
  • competitor-sov.json — the share-of-voice leaderboard (the business vs competitors across discovery queries), with is_client marking the business’s own row.

Full shapes: references/deep/data-contract.md.

Phase 4 — Bing, Cloudflare, Lighthouse inputs

  • Bing / Copilot (bing-analysis.json) — the AI-performance export shows which pages Copilot cites. status: "pending" when there’s no access yet (the sales-lead case).
  • Cloudflare AI bots (cloudflare-bots-6day.json) — a few days of AI-crawler request totals. Healthy crawl access means discoverability isn’t the bottleneck; authority/content is. status: "pending" when there’s no access.
  • Lighthouse (lighthouse-summary.json) — mobile SEO/perf/best-practices. Site speed is not an AI-visibility factor — it’s appendix hygiene. If the site sits behind a bot-challenge (e.g. Cloudflare), run Lighthouse via a route that isn’t challenged, or the scores will be wrong.

For a sales-lead / free audit, the placeholder status: "pending" files (as shipped in references/example-client/data/) are correct — chatot renders labelled “illustrative example” tables in their place.

Phase 5 — Build the report + FAQ worksheet

node scripts/chatot.mjs <client-dir> --out <client-dir>/output

Produces the HTML, PDF and Markdown report, plus faq-worksheet.md — one fillable section per research/comparison query no one is cited for. Those answers become FAQ blocks added to existing service pages (not a rebuild — pages already have their structure). This is the on-page content lever.

Phase 6 — QA and deliver

Read the report against the data: scorecard numbers match query-analysis.json; funnels show the right per-engine breakdown; the SOV leaderboard has the right competitors; the gap list matches the zero-score queries; the appendix shows pending (sales-lead) or real (retention) data. Then deliver the HTML + PDF + Markdown (+ the FAQ worksheet).

Two variants (config.json → variant)

  • retention — a managed client, monthly/quarterly. Bing/Cloudflare data is real; tone is progress-tracking (“what we’re doing”). Stat framing is a cadence.
  • sales-lead (default) — a cold prospect / free audit. Bing/Cloudflare show pending with labelled sample tables; the report carries a revenue-case ROI callout and a soft-deadline CTA; §06 is “what we’d build”. Never describe the work as trivial — it is expert-to-execute and ongoing-to-maintain, and a dollar quantification (procedure/matter value × plausible AI-referral uplift × conversion) is expected before it goes to a prospect.

Hard-won lessons

  1. Verify the domain — a typo nulls citations silently. Check the live site.
  2. Re-derive scoring from the raw response — don’t trust stored flags; watch boolean casing in any CSV round-trip.
  3. Most citations point to the website, not just the map listing — keep the thesis “local presence signals including the profile and the website”, not profile-only.
  4. The real on-page lever is FAQ depth, not “add galleries/photos” — verify the live site before recommending structural changes it already has.
  5. Never propose contacting the client’s customers/patients (reviews, third-party profiles) — keep the plan to what the agency controls.
  6. Confirm what’s already managed — don’t ask the client to “grant access” to things already run for them.
  7. Print consistency: fixed grid column counts, explicit colgroup widths, and print-color-adjust: exact — chatot’s HTML already does this; keep it if you edit the layout.
  8. Keep claims hedged — n=1 per query/engine; present a tracked baseline, not a fixed ranking.
  9. Lighthouse behind a bot-challenge = wrong scores — sanity-check against what the live site actually looks like.
  10. Gemini returns no structured competitors — the SOV leaderboard is effectively one engine; footnote it.
  11. Sample/mock tables use generic URLs ([example].com), never the client’s real domain, plus an inline “sample — not measured” note, so a skimmer never mistakes them for real data.