unify positions with the rest of the wiki entities, semantic search for all

Author: monneyboi

poliloom/CLAUDE.md

@@ -21,9 +21,8 @@ This document outlines the high-level architecture and strategy for the PoliLoom
 - **API:** FastAPI with MediaWiki OAuth 2.0 authentication
 - **Database:** PostgreSQL with SQLAlchemy ORM and Alembic migrations
 - **LLM Integration:** OpenAI API for structured data extraction
-- **Vector Search:** SentenceTransformers ('paraphrase-multilingual-MiniLM-L12-v2') + pgvector extension
+- **Search:** Meilisearch with OpenAI embeddings for hybrid search (keyword + semantic)
 - **Storage:** Google Cloud Storage (GCS) for dump processing (automatic gs:// path detection)
-- **PyTorch:** CPU-only in Docker (`uv sync --extra cpu`), GPU for development (`uv sync --extra cu128`)
 
 **Important:** Always use `uv` for running Python commands and managing dependencies.
 
@@ -44,7 +43,7 @@ Each Wikidata entity type has a dedicated class (`WikidataPolitician`, `Wikidata
 For entity-linked properties (OpenAI's 500 enum limit):
 
 1. **Free-form Extraction:** LLM extracts natural language descriptions
-2. **Vector Mapping:** Generate embeddings → similarity search top 100 → LLM maps to specific Wikidata entity or None
+2. **Entity Mapping:** Meilisearch hybrid search (keyword + semantic) → top 100 candidates → LLM maps to specific Wikidata entity or None
 
 ## **4. Core Functionality**
 
@@ -60,7 +59,7 @@ For entity-linked properties (OpenAI's 500 enum limit):
 - **Wikipedia Content:** Fetch and process linked articles
 - **LLM Extraction:** OpenAI structured data API for politician properties
 - **Conflict Detection:** Flag discrepancies between extracted and existing Wikidata values
-- **Similarity Search:** Match unlinked entities using embeddings
+- **Similarity Search:** Match unlinked entities using Meilisearch hybrid search
 
 ### **API Endpoints**
 
@@ -87,11 +86,11 @@ _Use `--help` for detailed command documentation._
 - Actions: **Accept** new extracted data (submit to Wikidata), **Reject** incorrect extracted data (soft delete), **Deprecate** existing statements (mark as deprecated in Wikidata)
 - Supports multiple users and threshold-based workflows
 
-### **Embedding Workflow**
+### **Search & Similarity**
 
-- Position/Location embeddings initially NULL during import
-- Generated separately in batch processing for optimal performance
-- Used for similarity search in two-stage extraction
+- All entities indexed to Meilisearch with labels during import
+- Meilisearch uses OpenAI embeddings for hybrid search (keyword + semantic)
+- Position entities use higher semantic ratio (0.8) for better matching
 
 ### **Conflict Handling**
 
@@ -163,9 +162,6 @@ uv run poliloom import-hierarchy --file ./dump.json
 uv run poliloom import-entities --file ./dump.json
 uv run poliloom import-politicians --file ./dump.json
 
-# Generate embeddings
-uv run poliloom embed-entities
-
 # Enrich politician data
 uv run poliloom enrich-wikipedia --id Q6279
 uv run poliloom enrich-wikipedia --limit 100
@@ -187,15 +183,15 @@ uv run poliloom garbage-collect
 
 - **Framework**: pytest with asyncio support
 - **Database**: PostgreSQL test database (port 5433)
-- **Mocking**: External APIs (OpenAI, sentence-transformers) mocked in `conftest.py`
+- **Mocking**: External APIs (OpenAI, Meilisearch) mocked in `conftest.py`
 - **Coverage Focus**: Entity classes, database models, core data pipeline
 - **Approach**: Minimal, behavior-focused testing. Test business logic and data transformations, not language mechanics (inheritance, type checking). Avoid over-engineering tests.
 
 ### **Key Patterns**
 
 - **Entity-Oriented Architecture**: Each Wikidata entity type has dedicated class
 - **Date Handling**: Store incomplete dates as strings ('1962', 'JUN 1982')
-- **Embedding Strategy**: NULL during import, batch-generated separately
+- **Search Indexing**: Entities indexed to Meilisearch during import, embeddings generated by Meilisearch
 - **Error Handling**: Comprehensive logging and graceful degradation
 
 ### **Pre-commit Configuration**

poliloom/Dockerfile

@@ -10,11 +10,11 @@ ENV UV_COMPILE_BYTECODE=1 \
 # Set working directory
 WORKDIR /app
 
-# Install all dependencies first with CPU-only PyTorch
+# Install all dependencies first
 RUN --mount=type=cache,target=/root/.cache/uv \
     --mount=type=bind,source=pyproject.toml,target=pyproject.toml \
     --mount=type=bind,source=uv.lock,target=uv.lock \
-    uv sync --frozen --no-install-project --no-dev --extra cpu
+    uv sync --frozen --no-install-project --no-dev
 
 # Copy package source and setup files
 COPY pyproject.toml ./
@@ -45,8 +45,8 @@ ENV PATH="/app/.venv/bin:$PATH"
 # Create non-root user
 RUN groupadd -r poliloom && useradd -r -g poliloom poliloom
 # Create cache directories with correct ownership
-RUN mkdir -p /var/cache/wikidata /var/cache/huggingface /var/cache/playwright && \
-    chown -R poliloom:poliloom /var/cache/wikidata /var/cache/huggingface /var/cache/playwright
+RUN mkdir -p /var/cache/wikidata /var/cache/playwright && \
+    chown -R poliloom:poliloom /var/cache/wikidata /var/cache/playwright
 
 # Install Playwright browsers as root, then fix ownership
 ENV PLAYWRIGHT_BROWSERS_PATH=/var/cache/playwright

poliloom/README.md

@@ -5,7 +5,8 @@ The Python backend for PoliLoom — processes Wikidata dumps, extracts politicia
 ## Requirements
 
 - Python 3.12+ with [uv](https://docs.astral.sh/uv/)
-- PostgreSQL with pgvector extension
+- PostgreSQL
+- Meilisearch
 - Linux or macOS (Windows not supported due to multiprocessing requirements)
 - OpenAI API key
 
@@ -41,9 +42,6 @@ make extract-wikidata-dump
 uv run poliloom import-hierarchy      # Build entity relationship trees
 uv run poliloom import-entities       # Import positions, locations, countries
 uv run poliloom import-politicians    # Import politicians
-
-# Generate embeddings for semantic search
-uv run poliloom embed-entities
 ```
 
 ### Extract politician data
@@ -74,7 +72,6 @@ API documentation available at http://localhost:8000/docs
 | `poliloom import-hierarchy`   | Build position/location hierarchy trees from Wikidata |
 | `poliloom import-entities`    | Import positions, locations, and countries            |
 | `poliloom import-politicians` | Import politicians linking to existing entities       |
-| `poliloom embed-entities`     | Generate vector embeddings for semantic search        |
 | `poliloom enrich-wikipedia`   | Extract politician data from Wikipedia using AI       |
 | `poliloom garbage-collect`    | Remove entities deleted from Wikidata                 |
 

poliloom/alembic/versions/b351d1ca5357_remove_embedding_column_from_positions.py

@@ -0,0 +1,65 @@
+"""remove embedding column from positions
+
+Revision ID: b351d1ca5357
+Revises: d6772e534c56
+Create Date: 2025-12-14 15:30:55.249491
+
+"""
+
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+import pgvector.sqlalchemy
+
+
+# revision identifiers, used by Alembic.
+revision: str = "b351d1ca5357"
+down_revision: Union[str, None] = "d6772e534c56"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    """Upgrade schema."""
+    # Drop the embedding reset trigger and function first
+    op.execute(
+        "DROP TRIGGER IF EXISTS wikidata_entity_name_change_trigger ON wikidata_entities"
+    )
+    op.execute("DROP FUNCTION IF EXISTS reset_embedding_on_name_change()")
+
+    # Drop the embedding column
+    op.drop_column("positions", "embedding")
+
+
+def downgrade() -> None:
+    """Downgrade schema."""
+    # Re-add the embedding column
+    op.add_column(
+        "positions",
+        sa.Column(
+            "embedding",
+            pgvector.sqlalchemy.vector.VECTOR(dim=384),
+            autoincrement=False,
+            nullable=True,
+        ),
+    )
+
+    # Re-create the embedding reset function and trigger
+    op.execute("""
+        CREATE OR REPLACE FUNCTION reset_embedding_on_name_change()
+        RETURNS TRIGGER AS $$
+        BEGIN
+            IF OLD.name IS DISTINCT FROM NEW.name THEN
+                UPDATE positions SET embedding = NULL WHERE wikidata_id = NEW.wikidata_id;
+            END IF;
+            RETURN NEW;
+        END;
+        $$ LANGUAGE plpgsql;
+    """)
+    op.execute("""
+        CREATE TRIGGER wikidata_entity_name_change_trigger
+            AFTER UPDATE ON wikidata_entities
+            FOR EACH ROW
+            EXECUTE FUNCTION reset_embedding_on_name_change();
+    """)

poliloom/poliloom/api/entities.py

@@ -6,7 +6,7 @@
 from sqlalchemy import select, func, and_, case
 
 from ..database import get_db_session
-from ..search import SearchService, get_search_service
+from ..search import SearchService
 from ..models import (
     Language,
     Country,
@@ -174,15 +174,15 @@ async def endpoint(
             description=f"Maximum number of {entity_name} to return",
         ),
         db: Session = Depends(get_db_session),
-        search_service: SearchService = Depends(get_search_service),
         current_user: User = Depends(get_current_user),
     ):
         f"""
         Search {entity_name} by name/label using semantic similarity.
 
         Returns matching {entity_name} ranked by relevance with hierarchy data.
         """
-        entity_ids = model_class.find_similar(q, db, search_service, limit=limit)
+        search_service = SearchService()
+        entity_ids = model_class.find_similar(q, search_service, limit=limit)
         if not entity_ids:
             return []
 

poliloom/poliloom/api/politicians.py

@@ -17,7 +17,7 @@
     enrich_batch,
     has_enrichable_politicians,
 )
-from ..search import SearchService, get_search_service
+from ..search import SearchService
 from ..models import (
     ArchivedPage,
     ArchivedPageLanguage,
@@ -250,15 +250,15 @@ async def search_politicians(
         default=50, le=100, description="Maximum number of politicians to return"
     ),
     db: Session = Depends(get_db_session),
-    search_service: SearchService = Depends(get_search_service),
     current_user: User = Depends(get_current_user),
 ):
     """
     Search politicians by name/label using semantic similarity.
 
     Returns matching politicians ranked by relevance with their properties.
     """
-    entity_ids = Politician.find_similar(q, db, search_service, limit=limit)
+    search_service = SearchService()
+    entity_ids = Politician.find_similar(q, search_service, limit=limit)
     if not entity_ids:
         return []