feat(phase-1): scraper + sanitizer + coverage probe with W26 quality report

[RyanAlberts]

What

Phase 1 PR #1: lands the data-quality floor before any LLM cost is incurred. First end-to-end probe on real W26 data revealed only 63.3% of the 196-company batch is analyzable from the upstream feed.

Closes #1.

Why

Per the user's directive: "don't hallucinate, ground each dashboard in data, acknowledge any companies that drop off due to data quality, and report a % of YC batch coverage metric." This PR makes coverage the headline metric and the dropped register the most-visible block on the dashboard — no quiet drops.

How

Module	Role
src/ycai/schemas.py	Pydantic models. Single source of truth.
src/ycai/scraper.py	yc-oss/api as the only sanctioned source per ADR 0001. Hard-fail on unreachable.
src/ycai/sanitizer.py	Defensive PII strip before disk/LLM. Idempotent.
src/ycai/coverage.py	Tier A/B/C classifier + dropped register.
src/ycai/verifier.py	Async link-checker (HEAD + GET fallback).
src/ycai/dashboard.py	Single-file HTML. Coverage headline + dropped register render before any chart.
src/ycai/cli.py	ycai run-coverage wires it all up.

W26 quality probe results

Batch: Winter 2026 (132 in upstream / 196 official)
Tier A (full):     120
Tier B (partial):  4    (website 4xx/5xx — kept with a flag)
Tier C (excluded): 8    (named in the dropped register)
Coverage of upstream:    93.9%
Coverage of YC official: 63.3%  ← headline

64 companies are missing because yc-oss/api is stale (last refreshed 2026-02-08, ~3 months before W26 closed). 8 are dropped for missing fields and named explicitly. 4 had dead websites.

Sanitized example: examples/output/dashboard-w26-2026-05-01.html. Full writeup: docs/QUALITY_REPORT_W26.md.

Test plan

• 41 tests pass (sanitizer, scraper, coverage, smoke). Network-free thanks to httpx MockTransport.
• make publish-check green (test fixtures with intentional fake credential patterns gated by inline pragmas + script exclusions).
• Real run on W26: produces dashboard.html, coverage.json, companies.csv. Verified manually.
• Hard-fail when yc-oss is unreachable: tested via 404 mock.
• PII strip is idempotent: tested.
• Sanitizer redacts before data hits disk: tested via cache_dir round-trip.

Anti-hallucination invariants this PR adds

• All numbers in the dashboard are computed in Python from validated rows. No LLM yet.
• Required-fields gate (description ≥80 chars, website starts with http/https, industry non-empty) before any company appears in charts.
• Every dropped company is named with a specific reason; no quiet exclusions.
• Coverage % is rendered against both upstream and (when known) YC-official count, so upstream staleness is visible.

Backlog spawned by this PR

• B004: tune MIN_DESCRIPTION_CHARS based on borderline rows.
• B005: name the 64 missing-from-upstream W26 companies (compare yc-oss slugs to a slug list discovered from /companies/<slug> profile pages).

Acceptance

• make validate-p0 green locally
• make publish-check green
• Pre-commit hooks all pass
• BACKLOG groomed
• LLM path: not applicable (no LLM in this PR; first invariants land in PR PR #2 — classifier + researcher with anti-hallucination Layer 1 #2)
• Real-data smoke run captured in examples/output/

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request implements Phase 1 of the project, introducing a scraper for yc-oss/api, a PII sanitizer, an async link verifier, and a data-quality coverage probe. It adds a Typer-based CLI with a run-coverage command that generates machine-readable JSON reports and interactive HTML dashboards. Key logic includes tier-based classification of companies to identify data gaps and a 'dropped register' for transparency. Feedback focuses on improving type safety in the CLI, refining HTML/JSON escaping in the dashboard generator, and ensuring dynamic rendering of company tiers in the output tables.

[gemini-code-assist]

The industry rows table currently hardcodes the 'Tier A' badge. Since the report includes both Tier A and Tier B companies, the badge should dynamically reflect the actual tier of each company.

Suggested change

	industry_rows = "\n".join(
	f"<tr><td><code>{_escape(c.slug)}</code></td>"
	f"<td>{_escape(c.name)}</td>"
	f"<td>{_escape(c.industry)}</td>"
	f'<td><span class="badge tier-A">A</span></td></tr>'
	for c in keepers
	)
	slug_to_tier = {r.slug: r.tier for r in coverage.records}
	industry_rows = "\n".join(
	f"<tr><td><code>{_escape(c.slug)}</code></td>"
	f"<td>{_escape(c.name)}</td>"
	f"<td>{_escape(c.industry)}</td>"
	f'<td><span class="badge tier-{slug_to_tier[c.slug]}">{slug_to_tier[c.slug]}</span></td></tr>'
	for c in keepers
	)

[gemini-code-assist]

HTML-escaping the JSON string inside a <script type="application/json"> tag breaks the JSON format (e.g., by replacing quotes with "). This will cause client-side parsing to fail. To safely embed JSON in HTML, you should only escape the sequence </ to prevent the browser from prematurely closing the script tag.

Suggested change

	raw_data_json=_escape(json.dumps(raw, default=str)),
	raw_data_json=json.dumps(raw, default=str).replace("</", "<\\/"),

[gemini-code-assist]

The csv module should be imported at the top level. Additionally, importing RawCompany here allows for better type hinting in the _write_csv function.

Suggested change

	import logging
	from datetime import UTC, datetime
	from pathlib import Path
	def app() -> None:
	"""Entry shim. Replaced with a Typer app in Phase 1."""
	print("yc-ai-pulse: Phase 0 scaffold. CLI lands in Phase 1.")
	import typer
	from rich.console import Console
	from rich.table import Table
	from ycai import __version__
	from ycai.coverage import compute_coverage, coverage_summary_lines
	from ycai.dashboard import render as render_dashboard
	from ycai.scraper import UpstreamError, fetch_batch, upstream_age_hours
	from ycai.verifier import check_urls, split_by_status
	import csv
	import logging
	from datetime import UTC, datetime
	from pathlib import Path
	import typer
	from rich.console import Console
	from rich.table import Table
	from ycai import __version__
	from ycai.coverage import compute_coverage, coverage_summary_lines
	from ycai.dashboard import render as render_dashboard
	from ycai.schemas import RawCompany
	from ycai.scraper import UpstreamError, fetch_batch, upstream_age_hours
	from ycai.verifier import check_urls, split_by_status

[gemini-code-assist]

Specifying list[RawCompany] instead of the generic list improves type safety. The local csv import can be removed as it is now handled at the top level.

Suggested change

	def _write_csv(companies: list, path: Path) -> None:
	"""Tiny CSV writer that doesn't pull in pandas just for serialization."""
	import csv
	def _write_csv(companies: list[RawCompany], path: Path) -> None:
	"""Tiny CSV writer that doesn't pull in pandas just for serialization."""

[gemini-code-assist]

Add import html to the top-level imports to support more robust HTML escaping.

Suggested change

	import json
	from collections import Counter
	from pathlib import Path
	from ycai.schemas import BatchCoverage, CoverageTier, RawCompany
	import html
	import json
	from collections import Counter
	from pathlib import Path
	from ycai.schemas import BatchCoverage, CoverageTier, RawCompany

[gemini-code-assist]

Using html.escape is preferred over a manual implementation as it is more comprehensive and follows standard practices.

Suggested change

	def _escape(text: str) -> str:
	return text.replace("&", "&amp;").replace("<", "&lt;").replace(">", "&gt;").replace('"', "&quot;")
	def _escape(text: str) -> str:
	return html.escape(text)