Simplified source scraping with campaigns

[monneyboi]

Overview

Simplified approach for adding government sources (parliaments, ministries, etc.) as data sources. Instead of scraper configs, we send index pages directly to the model (like we already do with detail pages). No pagination support.

Architecture

Refactor WikipediaLink → Source

Generalize the current WikipediaLink model to handle any source type:

class SourceType(enum.Enum):
    INDEX = "INDEX"      # List page with links to politicians
    DETAIL = "DETAIL"    # Individual politician page

class Source(Base):
    id: UUID
    url: str
    source_type: SourceType
    
    # Optional Wikipedia-specific fields
    wikipedia_project_id: str | None  # FK to WikipediaProject (NULL for non-Wikipedia)
    politician_id: UUID | None  # FK to Politician (NULL for INDEX sources)
    
    # Campaign association
    campaign_id: UUID | None  # FK to Campaign (NULL for Wikipedia links)
    
    # Relationships
    languages: list[Language]  # via SourceLanguage link table

class SourceLanguage(Base):
    """Link table between sources and language entities (many-to-many)."""
    source_id: UUID  # FK to Source, PK
    language_id: str  # FK to Language entity, PK

Sources can have multiple languages (Wikipedia projects can have multiple LANGUAGE_OF_WORK relations in Wikidata). The SourceLanguage link table mirrors the existing ArchivedPageLanguage pattern.

Campaign Model

Campaigns group index sources with metadata for batch processing:

class Campaign(Base):
    id: UUID
    name: str
    country_id: str | None  # FK to Country entity (optional filter)
    position_ids: list[str]  # Array of Position QIDs (optional filter)
    created_at: datetime
    
    # Relationships
    sources: list[Source]  # INDEX sources belonging to this campaign

Language on Source vs ArchivedPage

Move language association from ArchivedPage to Source:

• Language is known at source creation time (from Wikipedia project relations or user input)
• ArchivedPage becomes purely about content storage
• Simplifies the data model - source metadata stays with source
• Preserves support for multiple languages per source

Workflow

Index Page Processing

1. Create Campaign with index URLs as Source records (source_type=INDEX)
2. Run campaign:
   • Fetch each index URL via Playwright → ArchivedPage
   • Send rendered HTML to LLM with prompt: "Extract all politician detail page URLs from this index"
   • Create new Source records (source_type=DETAIL) for extracted URLs
   • Link detail sources to campaign's country/positions for filtering

Detail Page Processing

Detail pages go through existing enrichment pipeline:

• Triggered via enrich-wikipedia CLI or API
• Same LLM extraction for politician properties
• Same evaluation workflow

Supersedes #109

[monneyboi]

Updated Architecture: Split Source Models

After investigation, we've decided against a unified Source model. Instead, we'll use separate models for Wikipedia and campaign sources.

Rationale

A unified Source table creates problems:

• Nullable columns everywhere with complex conditional constraints
• Language denormalization for Wikipedia sources (copying from WikipediaProject)
• Mixed semantics - same model means different things depending on which fields are NULL
• Awkward queries requiring source_type checks or NULL pattern matching

New Schema

WikipediaSource (renamed from WikipediaLink, schema unchanged)
├── id, politician_id (required), wikipedia_project_id (required), url
└── UNIQUE(politician_id, wikipedia_project_id)

Campaign
├── id, name, country_id (nullable)

CampaignPosition (link table)
├── campaign_id, position_id

CampaignSource
├── id, campaign_id (required), url, politician_id (nullable)
└── politician_id NULL = index page, NOT NULL = detail page

ArchivedPage
├── id
├── wikipedia_source_id (nullable FK)
├── campaign_source_id (nullable FK)
├── url (reference URL - permanent oldid for Wikipedia, original for campaigns)
├── content_hash, fetch_timestamp
└── CHECK: exactly one source FK must be set

ArchivedPageLanguage (unchanged)
├── archived_page_id, language_id

Key Decisions

1. No SourceType enum - CampaignSource.politician_id IS NULL implies index page

2. Keep ArchivedPageLanguage - language is a property of the fetched snapshot:

   • Wikipedia: copy from WikipediaProject → LANGUAGE_OF_WORK at fetch time
   • Campaign: auto-detect from HTML at fetch time (future)

3. Unified language filtering: Property → ArchivedPage → ArchivedPageLanguage works for both source types

4. ArchivedPage.url for display - stores best reference URL (oldid for Wikipedia, original for campaigns), making ArchivedPage self-contained for display without joining to source tables. Replaces permanent_url field.