feat: implement seo drift monitoring system for baseline capture and change detection

Author: vikasswebdev

HOW_TO_USE.md

@@ -0,0 +1,153 @@
+# OpenSEO CLI Documentation & Usage Guide 🔍
+
+OpenSEO is a production-quality, provider-agnostic command-line tool built to help developers, marketers, and SEO professionals audit, analyze, and monitor search engine visibility using Large Language Models (LLMs) and local diagnostic checks.
+
+---
+
+## 🚀 Installation & Local Setup
+
+### 1. Standard Virtual Environment setup
+If you are developing locally from the source, create and activate a virtual environment:
+
+```powershell
+# Create environment
+python -m venv .venv
+
+# Activate environment (Windows PowerShell)
+.venv\Scripts\Activate.ps1
+
+# Activate environment (macOS/Linux)
+source .venv/bin/activate
+```
+
+### 2. Install OpenSEO in Editable Mode (Required for development)
+To ensure the `seo` CLI command references your local workspace changes (`src/` directory), install the package in **editable mode**:
+
+```bash
+pip install -e ".[all]"
+```
+*(Optionally run `playwright install chromium` if you wish to use javascript rendering features)*
+
+---
+
+## 🛠 Command Reference & Usage
+
+Run `seo --help` to see all available commands. Below is a detailed breakdown of each command and how to use it:
+
+---
+
+### 1. Setup & Diagnostics
+
+#### `seo init`
+Initializes OpenSEO with an interactive setup wizard that guides you through selecting default providers, models, caches, and output directory paths. Configurations are stored globally in `~/.openseo/config.json`.
+
+```bash
+seo init
+```
+
+#### `seo doctor`
+Runs a health check on your environment to verify API keys, cache database status, playwritght dependencies, and internet connectivity.
+
+```bash
+seo doctor
+```
+
+#### `seo provider`
+Lists available LLM providers, sets API credentials, and overrides default models.
+
+```bash
+# List all providers
+seo provider list
+
+# Set API key for a provider
+seo provider set-key openai sk-proj-...
+seo provider set-key gemini AIzaSy...
+
+# Set active provider and model
+seo provider use gemini --model gemini/gemini-1.5-flash
+```
+
+---
+
+### 2. Analysis & Auditing
+
+#### `seo audit`
+Crawls a website and runs rule-based technical checks combined with LLM analysis.
+
+```bash
+# Run a quick audit on a single URL
+seo audit https://example.com
+
+# Audit pages found in the sitemap index with custom depth
+seo audit https://example.com --sitemap-only --max-pages 20
+
+# Generate a complete PDF scorecard report inside the results/ folder
+seo audit https://example.com --report
+```
+
+#### `seo content`
+Analyzes content relevance against target keywords, checks search experience guidelines, and performs NLP/QRG diagnostics.
+
+```bash
+# Run LLM-based content keyword gap audit
+seo content https://example.com/blog/python-tutorial --keyword "python tutorial"
+
+# Run local Quality Rater Guidelines (QRG) checks (AI pattern, filler, repetition)
+seo content https://example.com/blog/python-tutorial --quality
+```
+
+#### `seo schema`
+Generates structured Schema.org JSON-LD blocks for search eligibility.
+
+```bash
+# Generate schema recommendations using page crawling + LLM analysis
+seo schema https://example.com/blog/post --type article
+
+# Interactively generate predefined high-leverage schema templates
+seo schema --template profile
+seo schema --template discussion
+seo schema --template order
+seo schema --template reservation
+```
+
+---
+
+### 3. SEO Drift & Regression Monitoring (`seo drift`)
+
+Allows capturing a technical baseline state of page structures and comparing current states to check for unintended deployments/code regressions (e.g., losing noindex directives, removing canonical tags, or deleting Schema).
+
+#### Capture a Baseline Snapshot
+```bash
+seo drift baseline https://example.com/pricing
+```
+
+#### Compare Current State to Baseline
+```bash
+# Compare against the latest captured baseline
+seo drift compare https://example.com/pricing
+
+# Compare against a specific baseline ID
+seo drift compare https://example.com/pricing --baseline-id 4
+
+# Output diff results as raw JSON
+seo drift compare https://example.com/pricing -o json
+```
+
+#### View Drift History
+```bash
+seo drift history https://example.com/pricing
+```
+
+#### Generate HTML Drift Reports
+```bash
+# Creates a self-contained color-coded HTML report (seo-drift-report.html)
+seo drift report https://example.com/pricing
+```
+
+---
+
+### 4. Utilities
+
+- `seo sitemap <url>`: Fetches and parses standard XML sitemaps to check schema compliance.
+- `seo robots <url>`: Audits site `robots.txt` configuration and maps user-agent blocks.
+- `seo keywords <topic>`: Generates keyword clustering and search-intent outlines.

src/openseo/analyzers/__init__.py

@@ -10,6 +10,7 @@
 from openseo.analyzers.technical import analyze_page_technical, analyze_site_technical
 from openseo.analyzers.duplicates import detect_duplicate_clusters
 from openseo.analyzers.link_graph import build_link_graph
+from openseo.analyzers.qrg import analyze_qrg
 
 __all__ = [
     "analyze_title",
@@ -23,4 +24,5 @@
     "analyze_site_technical",
     "detect_duplicate_clusters",
     "build_link_graph",
+    "analyze_qrg",
 ]

src/openseo/analyzers/qrg.py

@@ -0,0 +1,207 @@
+"""
+QRG-aligned content quality detector for OpenSEO.
+
+Ported from claude-seo's content_quality.py script.
+Scores text against Google's January 23, 2025 Quality Rater Guidelines update:
+  - §4.6.5 Scaled content abuse (using low-effort templates or AI generators)
+  - §4.6.6 MC (Main Content) copied or AI-generated without value
+  - §4.6 Filler content (padding phrases, low information density)
+"""
+
+from __future__ import annotations
+
+import re
+from collections import Counter
+from typing import Iterable, TypedDict
+
+
+class QRGAnalysisResult(TypedDict):
+    filler_score: int
+    ai_pattern_score: int
+    information_density: float
+    repetition_score: int
+    overall_quality: int
+    flags: list[str]
+    matches: dict[str, list[str]]
+    tokens: int
+    unique_tokens: int
+
+
+# Padding / filler phrases that QRG §4.6 flags as "little-to-no value".
+FILLER_PHRASES: tuple[str, ...] = (
+    "it's important to note that",
+    "in this article, we'll explore",
+    "in this article we will explore",
+    "in today's fast-paced world",
+    "in today's digital age",
+    "in today's competitive landscape",
+    "needless to say",
+    "at the end of the day",
+    "when it comes to",
+    "when all is said and done",
+    "in the realm of",
+    "in the world of",
+    "the bottom line is",
+    "without further ado",
+    "first and foremost",
+    "last but not least",
+    "for what it's worth",
+    "it goes without saying",
+    "as we all know",
+    "the truth is that",
+    "the fact of the matter is",
+    "more often than not",
+    "let's dive in",
+    "let's dive into",
+    "let's take a closer look",
+    "let's take a deeper look",
+)
+
+
+# LLM-typical phrasings (Wikipedia AI Cleanup catalogue, CC BY-SA 4.0;
+# also used by ivankuznetsov/claude-seo, MIT).
+AI_PATTERNS: tuple[str, ...] = (
+    "delve into",
+    "delve deeper into",
+    "in the ever-evolving",
+    "ever-evolving landscape",
+    "ever-changing landscape",
+    "in the dynamic landscape",
+    "navigating the",
+    "navigate the complexities",
+    "tapestry of",
+    "rich tapestry",
+    "intricate tapestry",
+    "embark on a journey",
+    "embarking on this",
+    "a testament to",
+    "a beacon of",
+    "the cornerstone of",
+    "a cornerstone of",
+    "at the heart of",
+    "at its core",
+    "in essence,",
+    "in conclusion,",
+    "ultimately,",
+    "moreover,",
+    "furthermore,",
+    "however, it's worth noting",
+    "it's worth noting that",
+    "by leveraging",
+    "leverage the power of",
+    "leveraging the power of",
+    "harness the power of",
+    "unlock the potential",
+    "unlock the full potential",
+    "the realm of possibilities",
+    "open up a world of",
+    "a world of possibilities",
+    "elevate your",
+    "transform your",
+    "revolutionize the way",
+    "game-changer",
+    "game-changing",
+    "cutting-edge",
+    "state-of-the-art",
+    "in summary,",
+    "to summarize,",
+    "to put it simply,",
+    "in a nutshell,",
+)
+
+
+TOKEN_RE = re.compile(r"[A-Za-z][A-Za-z'\-]*")
+NUMBER_RE = re.compile(r"\b\d+(?:[.,]\d+)?(?:%|st|nd|rd|th)?\b")
+# Capitalised multi-word names: rough proper-noun heuristic. Two or more
+# capitalised tokens in a row count as one entity.
+ENTITY_RE = re.compile(r"\b(?:[A-Z][a-z]+(?:\s+[A-Z][a-z]+)+)\b")
+
+
+def _count_phrase_hits(text: str, patterns: Iterable[str]) -> list[str]:
+    """Return the patterns that appear at least once in text (case-insensitive)."""
+    lowered = text.lower()
+    return [p for p in patterns if p in lowered]
+
+
+def _repetition_score(tokens: list[str]) -> float:
+    """Bigram repetition: fraction of bigrams that recur more than once."""
+    if len(tokens) < 4:
+        return 0.0
+    bigrams = [f"{tokens[i]} {tokens[i+1]}" for i in range(len(tokens) - 1)]
+    counts = Counter(bigrams)
+    repeated = sum(1 for v in counts.values() if v > 1)
+    return repeated / max(1, len(counts))
+
+
+def analyze_qrg(text: str) -> QRGAnalysisResult:
+    """Score a body of text against the QRG quality heuristics."""
+    if not text or not text.strip():
+        return {
+            "filler_score": 0,
+            "ai_pattern_score": 0,
+            "information_density": 0.0,
+            "repetition_score": 0,
+            "overall_quality": 0,
+            "flags": ["empty-input"],
+            "matches": {"filler": [], "ai_patterns": []},
+            "tokens": 0,
+            "unique_tokens": 0,
+        }
+
+    tokens = [t.lower() for t in TOKEN_RE.findall(text)]
+    n_tokens = len(tokens)
+    unique = len(set(tokens))
+
+    filler_hits = _count_phrase_hits(text, FILLER_PHRASES)
+    ai_hits = _count_phrase_hits(text, AI_PATTERNS)
+
+    # Density: entities + numbers per 100 tokens. A typical high-density
+    # article lands at ~5+; a generic filler post lands at <2.
+    entities = len(ENTITY_RE.findall(text))
+    numbers = len(NUMBER_RE.findall(text))
+    density_per_100 = (entities + numbers) * 100.0 / max(1, n_tokens)
+    information_density = min(1.0, density_per_100 / 10.0)
+
+    rep = _repetition_score(tokens)
+    rep_score = int(round(rep * 100))
+
+    # Scale to per-1000 tokens so the score is comparable across page lengths.
+    scale = max(1.0, n_tokens / 1000.0)
+    filler_per_kt = len(filler_hits) / scale
+    ai_per_kt = len(ai_hits) / scale
+
+    filler_score = min(100, int(round(filler_per_kt * 25)))
+    ai_pattern_score = min(100, int(round(ai_per_kt * 15)))
+
+    flags: list[str] = []
+    if filler_score >= 50:
+        flags.append("filler")
+    if ai_pattern_score >= 40:
+        flags.append("ai-patterns")
+    if information_density < 0.20:
+        flags.append("low-density")
+    if rep_score >= 30:
+        flags.append("repetitive")
+    if n_tokens < 300:
+        flags.append("thin-content")
+
+    # Composite: invert penalty signals, weight by impact.
+    overall = (
+        (100 - filler_score) * 0.25
+        + (100 - ai_pattern_score) * 0.25
+        + information_density * 100 * 0.25
+        + (100 - rep_score) * 0.15
+        + min(100, n_tokens / 10.0) * 0.10  # length bonus capped at 1000 tokens
+    )
+
+    return {
+        "filler_score": filler_score,
+        "ai_pattern_score": ai_pattern_score,
+        "information_density": round(information_density, 3),
+        "repetition_score": rep_score,
+        "overall_quality": int(round(overall)),
+        "flags": flags,
+        "matches": {"filler": filler_hits, "ai_patterns": ai_hits},
+        "tokens": n_tokens,
+        "unique_tokens": unique,
+    }

src/openseo/app.py

@@ -59,7 +59,7 @@ def create_app() -> typer.Typer:
     )
 
     # ── Register Built-in Commands ────────────────────────────────────────────
-    from openseo.commands import audit, config_cmd, content, doctor, init, keywords, provider, robots, schema, sitemap
+    from openseo.commands import audit, config_cmd, content, doctor, init, keywords, provider, robots, schema, sitemap, drift
 
     init.register(app)
     config_cmd.register(app)
@@ -71,6 +71,7 @@ def create_app() -> typer.Typer:
     doctor.register(app)
     sitemap.register(app)
     robots.register(app)
+    drift.register(app)
 
     # ── Version Command ───────────────────────────────────────────────────────
     @app.command("version")