feat(phase-1): LLM enrichment with anti-hallucination Layer 1

[RyanAlberts]

What

Phase 1 PR #2: LLM-driven classification of YC companies on top of PR #1's coverage probe. Three backends (Agent SDK default for subscription, Anthropic API for pay-per-token, Mock for tests) all return validated CompanyAnalysis objects with full Layer 1 anti-hallucination enforcement.

Closes #2.

Why

PR #1 surfaces which companies we can analyze. PR #2 actually classifies them — industry, AI capability, tech stack, OSS posture. The classification has to be cheap, schema-strict, and resistant to model fabrication; otherwise the eventual VC report is no better than vibes.

How

Module	Role
schemas.CompanyAnalysis	Pydantic source of truth. Closed-set enums for every classified field. sources field has min_length=1 — pydantic rejects empty.
classifier.map_industry	Deterministic prefill from yc-oss tags → Industry enum, no LLM.
researcher.Backend	Abstract protocol. Three implementations.
researcher.analyze	The pipeline. Schema enforcement, source-URL grounding, two-pass cross-check, sentinel on failure.
cli.run_coverage --enrich	Opt-in flag. --enrich-limit for smoke runs. --api-key overrides subscription mode. Rich progress UI.

Anti-hallucination Layer 1 — what's guaranteed

1. Schema-enforced output. Pydantic CompanyAnalysis rejects any model response missing required fields, with empty sources, or with invalid enum values.
2. Source-URL grounding. _validate_sources rejects citations that don't start with company.website or company.url. Fabricated URLs downgrade the row to confidence=low.
3. Two-pass cross-check. confidence=medium rows trigger a second independent classification pass. Disagreement on industry_primary or oss_posture downgrades to low. Agreement upgrades to high.
4. Sentinel low-confidence row on any failure. Parse error, validation error, source rejection, network timeout — all produce a clearly-labeled sentinel that survives in the CSV but is excluded from charts. No silent drops.
5. PII stripped before prompt. strip_pii runs over name / one_liner / long_description before they reach the model.

Live smoke run on real W26 data

Ran ycai run-coverage --enrich --enrich-limit 5 via subscription. 5 companies enriched in 39 seconds, ~free.

Slug	Confidence	AI capability	Notable
bidflow	high	rag, nlp-classic, agents	Real Estate / Construction
travo	high	rag, data-pipeline, agents	Real Estate / Construction
gru.space	high	no-ai	Industrials — model correctly identified this is not an AI company
autumn-ai	high	rag, data-pipeline, nlp-classic, agents	B2B SaaS
velum-labs	low (sentinel)	unclear	Schema validation failed — pipeline correctly fell through to the sentinel rather than serving an unverified analysis

Every cited source URL came from the inputs (website or YC profile). Zero fabricated citations.

Sample output checked in: examples/output/analyses-w26-smoke-2026-05-01.json.

Test plan

• 78 tests (40 new). All deterministic — no real LLM calls in CI.
• Trap-resistance suite: every fixture in hallucination_traps.json lands at confidence=low when the backend returns nothing OR when it returns a fabricated source URL.
• PII redaction-in-prompt verified by capturing the prompt sent to the backend.
• API key no-leakage test asserts --api-key value never appears in repr, public attrs, or any string representation of the backend.
• make publish-check green.
• Mypy --strict green.
• Live smoke run on subscription documented above.

Backlog spawned

• B006: track schema-validation failure rate during enrichment. Smoke run had 1/5 (20%) parse failures — likely rationale field exceeded the 400-char limit. Measure across the full batch and tune.
• B007: tech-stack and OSS posture come back as unknown for most companies because the model only sees the YC long_description. A depth=1 website crawl before classification would dramatically improve signal density.

Acceptance

• make validate-p0 green
• make publish-check green
• Pre-commit hooks pass
• LLM-path invariants preserved (the five listed above)
• Live smoke run captured

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request implements Phase 1 PR #2, introducing LLM-based enrichment for company data with a focus on anti-hallucination. It adds a researcher module supporting multiple backends (Claude Agent SDK, Anthropic API, and Mock), a deterministic industry classifier, and updated CLI functionality with an --enrich flag. Feedback focuses on hardening the anti-hallucination guards, specifically addressing a prefix-based URL validation vulnerability and potential crashes from malformed LLM responses. Other suggestions include improving logging for backend failures, replacing magic numbers with constants, and aligning Pydantic schema constraints with the LLM prompt instructions.

[gemini-code-assist]

The current implementation of _looks_like_input_url is vulnerable to prefix-based hallucinations. For example, if the allowed website is https://company.com, a source URL like https://company.com.attacker.com would be accepted because it starts with the allowed prefix. This bypasses the anti-hallucination grounding layer.

def _looks_like_input_url(url: str, company: RawCompany) -> bool:
    """Defense against hallucinated source URLs — a source must come from the inputs."""
    allowed = [company.website, company.url]
    for base in allowed:
        if not base:
            continue
        base = base.rstrip("/")
        # Ensure the URL is exactly the base or a sub-path/query/fragment
        if url == base or any(url.startswith(base + c) for c in ("/", "?", "#")):
            return True
    return False

[gemini-code-assist]

The json.loads call can return types other than a dictionary (e.g., a list or a string if the model outputs valid JSON of those types). Calling .setdefault() on a non-dict object will raise an AttributeError, causing the pipeline to crash instead of falling back to the sentinel row.

    try:
        payload = json.loads(match.group(0))
    except json.JSONDecodeError:
        return None
    if not isinstance(payload, dict):
        return None
    payload.setdefault("slug", slug)  # the model sometimes drops the slug

[gemini-code-assist]

Logging only the exception class name (exc.__class__.__name__) makes debugging difficult when a backend fails. It is better to include the full exception message to understand why the request failed (e.g., authentication error vs. rate limit).

Suggested change

	log.warning("AnthropicAPIBackend failure: %s", exc.__class__.__name__)
	log.warning("AnthropicAPIBackend failure: %s", exc)

[gemini-code-assist]

The semaphore limit of 8 is a magic number. It should be defined as a constant (e.g., MAX_CONCURRENT_ENRICHMENTS) at the module level to improve maintainability and make it easier to tune based on different backend rate limits.

[gemini-code-assist]

The prompt in researcher.py specifies that oss_evidence_url is mandatory if the posture is not 'closed', 'api-only', or 'unknown'. However, this constraint is not enforced in the Pydantic model. Adding a model validator would ensure that the Layer 1 anti-hallucination layer correctly rejects responses that violate this rule.