diff --git a/README.md b/README.md index 9799bbb..f9aab97 100644 --- a/README.md +++ b/README.md @@ -21,7 +21,7 @@ Search engines rank you in a list of links. Answer engines quote you inside a synthesized reply. Those are different games, and a page can win the first while losing the second: to be quoted, an answer engine has to fetch your page with its own crawler, read it without executing your JavaScript, and find a passage worth lifting. willaicite scores all three, one verdict per engine, and tells you exactly what to fix. -Point it at a URL and get a 0 to 100 score across eight weighted dimensions, with evidence on every line: the exact robots.txt rule that blocks a bot, the HTTP status a crawler actually receives, the sentence an engine could quote. There are no LLM calls, so the same input always produces the same score. +Point it at a URL and get a 0 to 100 score across nine weighted dimensions, with evidence on every line: the exact robots.txt rule that blocks a bot, the HTTP status a crawler actually receives, the sentence an engine could quote. There are no LLM calls, so the same input always produces the same score. Try it now at **[willaicite.com/audit](https://willaicite.com/audit)**, or run it locally. @@ -35,9 +35,9 @@ SEO tooling audits the path to a ranking. willaicite audits the path to a citati - **A verdict per engine, weighed by what it costs you.** "Is Googlebot allowed" is one question. willaicite asks it for a roster of AI crawler tokens and separates retrieval crawlers, where a block means that engine can never cite you, from training-only tokens, where blocking is a legitimate policy choice. - **Is there a liftable answer.** Engines quote a chunk verbatim. willaicite checks whether one exists: a self-contained answer near the top, headings phrased as the questions people actually ask, an FAQ ready to be lifted whole. You can rank first for a query and still contain nothing an engine can use. -## The eight dimensions +## The nine dimensions -Every audit scores the same eight dimensions, each weighted by how often it decides whether a citation happens. The overall score is the weighted average of the dimensions that could be verified. The weights follow the 2025–2026 replication literature, not the 2024 headline numbers: v1.3 added topical focus (high), promoted freshness to high, and demoted evidence density to medium. +Every audit scores the same nine dimensions, each weighted by how often it decides whether a citation happens. The overall score is the weighted average of the dimensions that could be verified. The weights follow the 2025–2026 replication literature, not the 2024 headline numbers: v1.3 added topical focus (high), promoted freshness to high, and demoted evidence density to medium. v1.4 added SEO foundation (low): the narrow slice of classic SEO hygiene that also decides which URL an engine attributes, kept at low weight because it is a hygiene multiplier, not a primary citation driver. | # | Dimension | Weight | What it measures | |---|---|---|---| @@ -49,6 +49,7 @@ Every audit scores the same eight dimensions, each weighted by how often it deci | 06 | **Structured data** | medium | JSON-LD (Article, FAQPage, Organization, Person) that helps Google AI Overviews and entity trust | | 07 | **Freshness** | high | Visible and machine-readable dates; a recent timestamp is one of the few content factors that consistently lifted citation odds in controlled 2026 testing | | 08 | **Entity & E-E-A-T** | medium | A nameable author, organization, and provenance an engine can attribute | +| 09 | **SEO foundation** | low | Canonical URLs that reference the page they are on, and titles/descriptions unique across pages, so citations and retrieval land on the right URL | Also checked, informational and unscored: `llms.txt` presence, well-formedness, and consistency with robots.txt. diff --git a/public/index.html b/public/index.html index 179c8c0..fb4216a 100644 --- a/public/index.html +++ b/public/index.html @@ -4,7 +4,7 @@ audit · willaicite - + @@ -224,7 +224,7 @@

Will AI engines cite this site?

-

A scored audit of retrieval-and-citation readiness for ChatGPT, Perplexity, Google AI Overviews and Claude: eight dimensions, exact evidence, no fabricated numbers.

+

A scored audit of retrieval-and-citation readiness for ChatGPT, Perplexity, Google AI Overviews and Claude: nine dimensions, exact evidence, no fabricated numbers.

@@ -249,7 +249,7 @@

Will AI engines cite this site?

-
Eight dimensions
+
Nine dimensions
Fix first, by impact per effort
diff --git a/site/src/pages/about.astro b/site/src/pages/about.astro index f567a1c..7b31eb7 100644 --- a/site/src/pages/about.astro +++ b/site/src/pages/about.astro @@ -14,6 +14,7 @@ const dims = [ { name: 'Structured data', weight: 2, asks: 'Does JSON-LD tell engines who wrote what, and when' }, { name: 'Freshness', weight: 3, asks: 'Is there a machine-readable date signal from the last 90 days' }, { name: 'Entity & E-E-A-T', weight: 2, asks: 'Can the author and organization behind the page be identified' }, + { name: 'SEO foundation', weight: 1, asks: 'Does the canonical reference the page it is on, and are titles and descriptions unique across pages' }, ]; const faqs = [ @@ -23,7 +24,7 @@ const faqs = [ }, { q: 'Why does the audit stop at ten requests?', - a: 'Courtesy to the site being audited. Ten requests are enough to score all eight dimensions (the page twice, robots.txt, the sitemap, an about page, the favicon and llms.txt), and a small fixed cap means an audit can never hammer the site it is inspecting. The audit honors robots.txt while it works.', + a: 'Courtesy to the site being audited. Ten requests are enough to score all nine dimensions (the page twice, robots.txt, the sitemap, an about page, the favicon and llms.txt), and a small fixed cap means an audit can never hammer the site it is inspecting. The audit honors robots.txt while it works.', }, ]; @@ -118,7 +119,7 @@ const schema = { retrieved and cited by AI answer engines”.

-

What do the eight dimensions measure?

+

What do the nine dimensions measure?

diff --git a/site/src/pages/index.astro b/site/src/pages/index.astro index 628ce5c..a88b630 100644 --- a/site/src/pages/index.astro +++ b/site/src/pages/index.astro @@ -18,6 +18,7 @@ const dimensions = [ { name: 'Structured data', weight: 'medium', note: 'Schema.org markup that helps Google AI Overviews and entity trust.' }, { name: 'Freshness', weight: 'high', note: 'Visible and machine-readable dates; a recent timestamp is one of the few factors that consistently lifted citation odds in controlled 2026 testing.' }, { name: 'Entity & E-E-A-T', weight: 'medium', note: 'A nameable author, organization and provenance an engine can attribute.' }, + { name: 'SEO foundation', weight: 'low', note: 'Self-referencing canonicals and unique titles/descriptions, so citations land on the right URL.' }, ]; // FAQ is single-sourced: this array renders the visible section AND the @@ -265,11 +266,11 @@ const schema = { - +
-

The eight dimensions

+

The nine dimensions

- Every audit scores the same eight dimensions, weighted by how often each one decides + Every audit scores the same nine dimensions, weighted by how often each one decides whether a citation happens. The weights follow the 2025–2026 replication literature, not the 2024 headline numbers.

diff --git a/src/audit.ts b/src/audit.ts index 22fc27e..ea63ec3 100644 --- a/src/audit.ts +++ b/src/audit.ts @@ -12,10 +12,11 @@ import { checkTopicalFocus } from './checks/topicalFocus.js'; import { checkEvidenceDensity } from './checks/evidenceDensity.js'; import { checkFreshness } from './checks/freshness.js'; import { checkEntityEeat } from './checks/entityEeat.js'; +import { checkSeoFoundation } from './checks/seoFoundation.js'; import { checkLlmsTxt } from './checks/llmsTxt.js'; import { overallScore, verdictFor, prioritize } from './score.js'; -export const VERSION = '1.3.0'; +export const VERSION = '1.4.0'; const MAX_PAGES = 10; const OWN_BOT_TOKEN = 'geo-audit'; @@ -188,6 +189,7 @@ export function buildResult(inputUrl: string, ctx: AuditContext, now: Date = new ['evidenceDensity', () => checkEvidenceDensity(ctx)], ['freshness', () => checkFreshness(ctx, now)], ['entityEeat', () => checkEntityEeat(ctx)], + ['seoFoundation', () => checkSeoFoundation(ctx)], ]; const dimensions: DimensionResult[] = checks.map(([key, run]) => { diff --git a/src/checks/answerReadiness.ts b/src/checks/answerReadiness.ts index 79d28f5..386520b 100644 --- a/src/checks/answerReadiness.ts +++ b/src/checks/answerReadiness.ts @@ -111,6 +111,25 @@ export function checkAnswerReadiness(ctx: AuditContext): DimensionResult { evidence.push({ status: 'warn', message: `${h1s.length} H1 headings found; topic signal is diluted` }); } + // 2b. Heading hierarchy (advisory, unscored): a skipped level (H2 → H4) + // breaks the outline chunkers use to attach body text to its section. + const skips: string[] = []; + for (let i = 1; i < headings.length; i++) { + if (headings[i].level > headings[i - 1].level + 1) { + skips.push(`H${headings[i - 1].level} → H${headings[i].level} at "${headings[i].text.slice(0, 50)}"`); + } + } + if (skips.length > 0) { + evidence.push({ status: 'warn', message: `heading hierarchy skips levels (${skips.slice(0, 3).join('; ')})` }); + recommendations.push({ + dimension: dim, + action: 'Fix skipped heading levels so the outline nests without gaps (H2 under H1, H3 under H2)', + why: 'Chunkers and outline parsers use heading levels to decide which section a passage belongs to; a skipped level detaches the section from its parent topic and weakens the retrieved chunk\'s context.', + impact: 1, + effort: 1, + }); + } + // 3. Question-formatted headings (0-25) const subHeadings = headings.filter((h) => h.level >= 2); const questionHeadings = subHeadings.filter((h) => QUESTION_START.test(h.text) || h.text.trim().endsWith('?')); diff --git a/src/checks/seoFoundation.ts b/src/checks/seoFoundation.ts new file mode 100644 index 0000000..3d69a91 --- /dev/null +++ b/src/checks/seoFoundation.ts @@ -0,0 +1,152 @@ +import type { AuditContext, PageData } from '../context.js'; +import type { DimensionResult, Evidence, Recommendation } from '../types.js'; +import { extractCanonical, extractTitle, findMeta } from '../html.js'; + +/** + * SEO foundation: the small subset of classic SEO hygiene that also decides + * which URL an AI engine attributes and cites. Deliberately narrow — presence + * of title/description/canonical is already scored in topical focus; this + * dimension checks the cross-cutting failure modes those single-page checks + * cannot see: a canonical that points AWAY from the audited URL (the citation + * lands on a different address), and titles/descriptions duplicated across + * pages (retrieval cannot tell the pages apart). Low weight: these are + * hygiene multipliers, not primary citation drivers in the 2025-2026 + * literature. + */ + +/** Comparable URL identity: scheme-insensitive, www-insensitive, trailing-slash-insensitive. */ +function urlKey(url: string, base?: string): string | null { + try { + const u = base ? new URL(url, base) : new URL(url); + return u.hostname.replace(/^www\./, '') + (u.pathname.replace(/\/+$/, '') || '/') + u.search; + } catch { + return null; + } +} + +export function checkSeoFoundation(ctx: AuditContext): DimensionResult { + const dim = 'SEO foundation'; + const pages: PageData[] = [ctx.target, ctx.homepage, ctx.aboutPage, ...ctx.extraPages].filter( + (p): p is PageData => Boolean(p?.html), + ); + + if (pages.length === 0) { + return { + key: 'seoFoundation', + name: dim, + weight: 1, + score: null, + evidence: [{ status: 'unverified', message: 'could not verify: no page HTML available' }], + recommendations: [], + }; + } + + const evidence: Evidence[] = []; + const recommendations: Recommendation[] = []; + // Each assessable part contributes 0-100; the score renormalizes over the + // parts that could actually be checked (matches the overall-score policy). + const parts: number[] = []; + + // 1. Canonical self-consistency on the audited page. Presence alone is + // scored in topical focus; here the href must resolve back to the page + // itself, else every consumer that honors canonicals attributes the content + // to a different URL than the one audited. + if (ctx.target?.html) { + const pageUrl = ctx.target.fetch.finalUrl ?? ctx.target.url; + const canonical = extractCanonical(ctx.target.html); + if (canonical === null) { + parts.push(30); + evidence.push({ status: 'warn', message: 'no canonical link on the audited page; parameter/www/slash variants can split its retrieval identity' }); + recommendations.push({ + dimension: dim, + action: 'Add a self-referencing to the audited page', + why: 'Engines and crawl pipelines consolidate duplicate URL variants onto the canonical; without one, citations and link equity can scatter across variants of the same page.', + impact: 1, + effort: 1, + }); + } else { + const canonKey = urlKey(canonical, pageUrl); + const pageKey = urlKey(pageUrl); + if (canonKey !== null && pageKey !== null && canonKey === pageKey) { + parts.push(100); + evidence.push({ status: 'pass', message: `canonical is self-referencing (${canonical})` }); + } else { + parts.push(0); + evidence.push({ + status: 'fail', + message: `canonical points away from the audited page: canonical "${canonical}" vs page "${pageUrl}"`, + }); + recommendations.push({ + dimension: dim, + action: 'Fix the canonical URL so it references the page it is on (or audit the canonical target instead)', + why: 'A canonical pointing elsewhere tells every consumer that honors it — including AI crawl pipelines — that this page is a duplicate: the cited/indexed URL becomes the canonical target, not this page.', + impact: 2, + effort: 1, + }); + } + } + } else { + evidence.push({ status: 'unverified', message: 'canonical consistency not assessable: audited page HTML unavailable' }); + } + + // 2 & 3. Title / meta-description uniqueness across the audited pages. + // Duplicated metadata makes distinct pages indistinguishable to a retriever + // matching a query against titles and snippets. + const assessUniqueness = ( + label: 'title' | 'meta description', + valueOf: (html: string) => string | null, + action: string, + why: string, + ) => { + const withValue = pages + .map((p) => ({ url: p.url, value: valueOf(p.html!)?.trim().toLowerCase() ?? null })) + .filter((p): p is { url: string; value: string } => Boolean(p.value)); + if (withValue.length < 2) { + evidence.push({ status: 'info', message: `${label} uniqueness not assessable (fewer than 2 audited pages carry a ${label})` }); + return; + } + const byValue = new Map(); + for (const p of withValue) byValue.set(p.value, [...(byValue.get(p.value) ?? []), p.url]); + const dupes = [...byValue.values()].filter((urls) => urls.length > 1); + if (dupes.length === 0) { + parts.push(100); + evidence.push({ status: 'pass', message: `${label}s are unique across the ${withValue.length} audited pages that have one` }); + } else { + parts.push(0); + const example = dupes[0]; + evidence.push({ + status: 'fail', + message: `duplicated ${label} across pages (e.g. ${example.slice(0, 3).join(' and ')})`, + }); + recommendations.push({ dimension: dim, action, why, impact: 2, effort: 1 }); + } + }; + + assessUniqueness( + 'title', + extractTitle, + 'Give every page a unique naming that page\'s specific topic', + 'The title is the primary surface a retriever matches a query against; pages sharing one title compete as duplicates instead of each winning its own queries.', + ); + assessUniqueness( + 'meta description', + (html) => findMeta(html, 'description'), + 'Write a distinct meta description per page', + 'Engines use the description as a snippet/retrieval signal; identical descriptions across pages erase the differences a retriever could use to pick the right page.', + ); + + evidence.push({ + status: 'info', + message: `scope note: only the ${pages.length} fetched page(s) were compared; presence/length of title, description and canonical are scored under topical focus & metadata`, + }); + + const score = parts.length > 0 ? Math.round(parts.reduce((a, b) => a + b, 0) / parts.length) : null; + return { + key: 'seoFoundation', + name: dim, + weight: 1, + score, + evidence, + recommendations, + }; +} diff --git a/src/html.ts b/src/html.ts index b84e63d..ffa3338 100644 --- a/src/html.ts +++ b/src/html.ts @@ -172,6 +172,19 @@ export function jsonLdTypes(nodes: Record<string, unknown>[]): string[] { return types; } +/** href of the first `<link rel="canonical">`, or null. */ +export function extractCanonical(html: string): string | null { + const tags = html.match(/<link\b[^>]*>/gi) ?? []; + for (const tag of tags) { + const rel = getAttr(tag, 'rel'); + if (rel && /(^|\s)canonical(\s|$)/i.test(rel)) { + const href = getAttr(tag, 'href'); + if (href) return href; + } + } + return null; +} + export interface LinkTag { href: string; text: string; diff --git a/tests/checks.test.ts b/tests/checks.test.ts index 2bc9b29..8584a91 100644 --- a/tests/checks.test.ts +++ b/tests/checks.test.ts @@ -7,6 +7,7 @@ import { checkTopicalFocus } from '../src/checks/topicalFocus.js'; import { checkEvidenceDensity } from '../src/checks/evidenceDensity.js'; import { checkFreshness } from '../src/checks/freshness.js'; import { checkEntityEeat } from '../src/checks/entityEeat.js'; +import { checkSeoFoundation } from '../src/checks/seoFoundation.js'; import { checkLlmsTxt } from '../src/checks/llmsTxt.js'; import { makeCtx, makePage, makeFetch, fixture, robotsCtxFrom, NOW } from './helpers.js'; import { parseRobots } from '../src/robots.js'; @@ -428,3 +429,66 @@ describe('checkLlmsTxt', () => { expect(r.evidence.some((e) => e.status === 'warn' && e.message.includes('soft-404'))).toBe(true); }); }); + +describe('checkSeoFoundation', () => { + const page = (url: string, title: string, desc: string | null, canonical: string | null) => + makePage( + url, + `<html><head><title>${title}${desc ? `` : ''}${canonical ? `` : ''}

Body text long enough to exist.

`, + ); + + it('passes a self-referencing canonical and unique metadata (good article ctx)', () => { + const r = checkSeoFoundation(makeCtx()); + expect(r.weight).toBe(1); + expect(r.score).toBe(100); + expect(r.evidence.some((e) => e.status === 'pass' && e.message.includes('self-referencing'))).toBe(true); + }); + + it('tolerates www/trailing-slash/scheme variants in the canonical', () => { + const r = checkSeoFoundation(makeCtx({ target: page('https://example.com/guide', 'Guide To Widgets', null, 'http://www.example.com/guide/') })); + expect(r.evidence.some((e) => e.status === 'pass' && e.message.includes('self-referencing'))).toBe(true); + }); + + it('fails a canonical pointing at a different URL with an attribution rec', () => { + const r = checkSeoFoundation(makeCtx({ target: page('https://example.com/guide', 'Guide To Widgets', null, 'https://example.com/other-page') })); + expect(r.evidence.some((e) => e.status === 'fail' && e.message.includes('points away'))).toBe(true); + expect(r.recommendations.some((rec) => rec.action.includes('Fix the canonical'))).toBe(true); + }); + + it('warns (partial credit) when no canonical exists', () => { + const r = checkSeoFoundation(makeCtx({ target: page('https://example.com/guide', 'Guide To Widgets', null, null) })); + expect(r.score).toBe(30); + expect(r.recommendations.some((rec) => rec.action.includes('self-referencing'))).toBe(true); + }); + + it('fails duplicated titles/descriptions across audited pages', () => { + const r = checkSeoFoundation( + makeCtx({ + target: page('https://example.com/guide', 'Acme Widgets', 'Same blurb everywhere.', 'https://example.com/guide'), + homepage: page('https://example.com/', 'Acme Widgets', 'Same blurb everywhere.', 'https://example.com/'), + }), + ); + expect(r.evidence.some((e) => e.status === 'fail' && e.message.includes('duplicated title'))).toBe(true); + expect(r.evidence.some((e) => e.status === 'fail' && e.message.includes('duplicated meta description'))).toBe(true); + expect(r.score).toBe(33); // canonical 100, title 0, description 0 + }); + + it('passes uniqueness when pages differ', () => { + const r = checkSeoFoundation( + makeCtx({ + target: page('https://example.com/guide', 'Widget Guide', 'How widgets work.', 'https://example.com/guide'), + homepage: page('https://example.com/', 'Acme Widgets Home', 'Widgets for every need.', null), + }), + ); + expect(r.score).toBe(100); + }); + + it('marks uniqueness unassessable with a single audited page', () => { + const r = checkSeoFoundation(makeCtx({ aboutPage: null })); + expect(r.evidence.some((e) => e.status === 'info' && e.message.includes('title uniqueness not assessable'))).toBe(true); + }); + + it('could not verify without any HTML', () => { + expect(checkSeoFoundation(makeCtx({ target: null, homepage: null, aboutPage: null })).score).toBeNull(); + }); +}); diff --git a/tests/fixtures/article-good.html b/tests/fixtures/article-good.html index b662e53..8af4d5a 100644 --- a/tests/fixtures/article-good.html +++ b/tests/fixtures/article-good.html @@ -9,7 +9,7 @@ - +
dimensionweightquestion it answers