Research notes
What the snapshot is measuring, and why.
Every check in the report comes from a vendor's own documentation, a standard, or a named study. 0 notes, 0 sources.
01June 2026 · CC BY 4.0
The AI Visibility Audit: A Field Guide for SEOs and GEOs
Stephen Burns, Web Intelligence Lead, Common Crawl Foundation. Why being in the crawl decides what AI knows, how CDN and WAF defaults block AI crawlers without anyone choosing to, and a five-check audit that uses only free tools.
- CCBot access: robots.txt, then the edge, with the CCBot user agent against a browser control
- Common Crawl index coverage: is the domain in the archive, and how recently
- Harmonic centrality: how close the domain sits to the core of the web graph, which sets crawl priority
- Structured data completeness on key pages
- Server-side rendering: does the content exist without JavaScript
- The snapshot runs check 1 and check 5 itself, reads checks 2 and 3 from audit.stephenburns.ai, and covers check 4 with the structured-data dimension. The guide’s blocking statistics, two-layer model (training vs. retrieval) and English-share figure are cited where the report uses them.
How we measure
Every snapshot check, how it is measured, and what it rests on.
Only documented and measured checks move the score. Heuristics are writing suggestions. Scores are a model argued from these documents, not a vendor standard.
Bot access (A, 22%)
| Check | How | Evidence | Source |
|---|
| /robots.txt readable; 5xx or no answer treated as a complete disallow | Fetched as a browser. A 4xx means no restrictions, as RFC 9309 states. | Documented | RFC 9309; four-layers note |
| Search crawlers allowed: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot | Rules evaluated for the page path, longest match wins. | Documented | OpenAI, Anthropic, Perplexity, Google crawler docs; bots note |
| Search crawlers get the page | Each documented user agent fetches the page from this server. A refusal while a browser gets 200 is an edge block; a dropped connection is retested once, alone. | Measured | Field guide check 1; four-layers note |
| Browser control | If a plain browser request is also refused, bot results are marked inconclusive, not blocked. | Measured | Field guide check 1 |
| Training crawlers refused at the edge (CCBot, GPTBot, ClaudeBot, Amazonbot, Meta-ExternalAgent) | Reported as a fix when robots.txt allows the bot and a CDN or firewall refuses it. Not scored: training opt-outs are a policy choice. | Measured | Field guide §3; Common Crawl FAQ |
| robots.txt cloaking | The file fetched with the CCBot user agent is compared with the browser copy. | Measured | Field guide §3 (managed robots.txt) |
| Capped at 8 | A 10 needs server logs verified against the vendors’ IP ranges, which no outside fetch can see. | Documented | Score note |
HTML received (H, 20%)
| Check | How | Evidence | Source |
|---|
| Words in the raw HTML | Scripts, styles and markup removed; under 15 words is a shell (score 0, overall capped at 2). | Measured | Vercel/MERJ 2024; Common Crawl FAQ (CCBot runs no JavaScript) |
| Served as HTML, at most one redirect, title and H1 in the first response | Read from the first response. | Measured | Score note |
| A passage that stands alone is in the raw HTML | The opening paragraph under each H2 or H3 is read from the raw HTML. One that names the brand or product, does not open on “we”, “it”, “this” or “yes”, and is a full sentence (12+ words) is a passage a retriever can lift: with the rules above, 10. None yet: 8, and the note shows the closest opener and what it lacks. | Measured | Score note (the answer in the first body); how-AI-reads note; four-layers note, passage layer; Anthropic contextual retrieval |
| Snippet controls | nosnippet or max-snippet:0 fails the page; max-snippet limits and data-nosnippet are noted. | Documented | Google AI features guide; bots note |
Citations (C, 18%)
| Check | How | Evidence | Source |
|---|
| Left blank | One fetch cannot see whether an answer names the site. The score is published as fetch-only, divided by the weights that were measured. | Documented | Score note; four-layers note |
Links and sitemap (L, 12%)
| Check | How | Evidence | Source |
|---|
| Sitemap found; this page listed with a lastmod | Sitemap: line in robots.txt, then /sitemap.xml. For an index, one child sitemap is opened to look for the page. 8 when listed with a lastmod; 5 when lastmod is missing or one copied date; 2 when no sitemap. | Measured | sitemaps.org; RFC 9309 (Sitemap record); score note |
| Not done in a snapshot | Opening sampled URLs and checking internal links, which the score note needs for 10. | Measured | Score note |
Dates (F, 12%)
| Check | How | Evidence | Source |
|---|
| A machine-readable date somewhere | <time datetime>, article meta tags, JSON-LD dates, sitemap lastmod, feed dates. None: 0. | Measured | Score note |
| Page date agrees with its sitemap lastmod | Same calendar day: 8. Disagree: 5. A future date: 0. A lastmod not in W3C Datetime: 5. | Measured | sitemaps.org; Google 2014 sitemaps and feeds post; score note |
Feed (R, 8%)
| Check | How | Evidence | Source |
|---|
| An RSS or Atom feed is linked, parses and dates its items | rel="alternate" links first, then the usual paths. Comment feeds are ignored. A linked feed that loops or errors is reported with the reason. | Measured | Google 2014 post; sitemaps.org; score note |
Structured data (S, 8%)
| Check | How | Evidence | Source |
|---|
| JSON-LD in the first HTML names the organisation or site | Absent: 3. Present without a name: 4. | Documented | Google structured-data guidelines; Google AI features guide (no special type required) |
| Stated names, prices and ratings are visible on the page | Each value is searched for in the visible text. All visible: 7. Any hidden: 0. | Documented | Google structured-data guidelines (markup must describe page content) |
Not scored
| Check | How | Evidence | Source |
|---|
| Common Crawl Web Graph rank and archive presence | Read from audit.stephenburns.ai, cached a day, credited. Explains the training layer; does not decide today’s citations. | Measured | Field guide checks 2–3; Common Crawl Web Graph |
| Time to first byte, inline JavaScript share | Measurements shown without a pass mark; no vendor publishes a limit. | Measured | Score note; field guide check 5 |
| Writing suggestions: title length, question headings, sections opening on “we” or “it”, an English edition | Heuristics, listed apart. They never count against the score. | Heuristic | How-AI-reads note; Anthropic contextual retrieval; field guide §3 |
Primary sources
The documents behind the checks.
- Stephen Burns, The AI Visibility Audit: A Field Guide for SEOs and GEOs, Common Crawl Foundation, June 2026 (CC BY 4.0)stephenburns.ai
- Common Crawl, Introducing the AI Visibility Auditcommoncrawl.org
- Common Crawl FAQ (CCBot, robots.txt, JavaScript, sitemaps)commoncrawl.org
- Common Crawl Web Graphs (harmonic centrality)commoncrawl.org
- Google, AI features and your websitedevelopers.google.com
- Google, General structured data guidelinesdevelopers.google.com
- OpenAI, Overview of OpenAI crawlersdevelopers.openai.com
- Anthropic, Does Anthropic crawl data from the web?support.claude.com
- Perplexity, Perplexity crawlersdocs.perplexity.ai
- Amazon, Amazonbotdeveloper.amazon.com
- Meta, Meta web crawlersdevelopers.facebook.com
- Apple, About Applebotsupport.apple.com
- RFC 9309, Robots Exclusion Protocolrfc-editor.org
- sitemaps.org protocolsitemaps.org
- Vercel and MERJ, The rise of the AI crawler, December 2024vercel.com
- Anthropic, Introducing Contextual Retrieval, 2024anthropic.com
Interviews, such as the Common Crawl and harmonic-centrality podcast with Stephen Burns and James Dooley, are read as reported speech and are not used as sources for any check.