feat(categorization): semantic embedding tier + richer LLM prompts

[gianniskotsas]

Summary

• Embedding tier between deterministic keywords and the LLM: pgvector cosine ANN over category anchor embeddings built from name + description + keywords. Accepts only when similarity ≥ 0.78 AND margin ≥ 0.06 over runner-up; otherwise defers to LLM.
• Richer LLM prompts: the model now sees each category's description and keyword hints (vendor lists like Jumbo / Cloudflare / Efood / IBKR). Previously only category names were sent — the curated keywords were invisible.
• Smaller LLM batches (50 → 15) so each transaction gets a fair share of attention and prompts have room for hints.
• Persisted metadata on transactions: categorization_confidence, categorization_method ('override' | 'deterministic' | 'embedding' | 'llm'), plus a cached embedding vector for retry idempotency.
• Seed script for the four example categories (Side Projects, Groceries, Internal Transfer, Food & Dining).

Architecture

Pipeline now has four tiers:
user override → deterministic keywords → embedding semantic match → LLM fallback

Files

• backend/app/services/category_embedding.py — new CategoryEmbeddingService
• backend/app/services/category_matcher.py — _format_category_list_with_hints helper; prompt updates; batch size default
• backend/tasks/post_import_pipeline.py — embedding tier integrated into _batch_categorize_transactions
• backend/app/models.py + frontend/lib/db/schema.ts — new columns
• frontend/lib/db/migrations/0018_categorization_embeddings.manual.sql — pgvector extension, columns, HNSW index
• backend/scripts/seed_category_keywords.py — pre-populate curated keyword lists

Deploy steps

1. pip install -r backend/requirements.txt (adds pgvector>=0.3.0)
2. Apply migration: psql $DATABASE_URL -f frontend/lib/db/migrations/0018_categorization_embeddings.manual.sql (or pnpm db:push)
3. python -m scripts.seed_category_keywords <user_id> to backfill keywords and build anchor embeddings
4. Next import run will use all four tiers and persist method + confidence per transaction

Test plan

• Apply migration on a staging DB; confirm vector extension + new columns exist
• Run seed script for a user; verify categories.embedding IS NOT NULL for the 4 seeded categories
• Import a CSV containing a mix of vendor variants (e.g. "ALBERT HEIJN 1234", "AH to go AMSTERDAM") and confirm: (a) high-similarity ones are assigned via embedding tier (categorization_method = 'embedding'), (b) ambiguous ones fall through to LLM
• Confirm persisted categorization_confidence roughly matches observed match quality
• Existing unit tests for match_categories_batch_llm still pass

🤖 Generated with Claude Code

---

Summary by cubic

Adds a semantic embedding tier and richer LLM prompts to improve categorization accuracy and reduce LLM calls. We also persist method, confidence, and embeddings for auditability.

• New Features

  • Embedding tier using pgvector cosine over category anchor embeddings; accept when similarity ≥ 0.78 and margin ≥ 0.06 over runner-up, else defer to LLM.
  • LLM prompts now include category descriptions and keyword hints (400-char truncation); batch size 50 → 15; integrated with the prompt-context budget and account context.
  • Pipeline: override → embedding → LLM, with accurate categorization_method; store categorization_confidence (0–100) and transaction embedding.
  • Embedding safety: vectors fixed at 1536 dims; pass dimensions only for text-embedding-3* and always length-check. Scripts: seed_category_keywords to add hints + rebuild anchors; backfill_transaction_embeddings to populate transactions.embedding only (ID-first, low-memory; positive-int flags).

• Migration

  • Add pgvector and run migration 0018: categories.embedding; transactions.embedding, categorization_confidence, categorization_method; HNSW cosine index.
  • Deps/infra: install pgvector>=0.3.0; use Postgres image pgvector/pgvector:pg16 in CI/Compose; Drizzle vector(1536) custom type; frontend sets embedding: null on new categories.
  • Seed and refresh: python -m scripts.seed_category_keywords <user_id> then rebuild anchors; optional backfill: python -m scripts.backfill_transaction_embeddings <user_id>.

^{Written for commit a395ca5. Summary will update on new commits.}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
syllogic	Ready	Preview, Comment	May 31, 2026 6:59pm

[cubic-dev-ai]

2 issues found across 8 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="backend/tasks/post_import_pipeline.py">

<violation number="1" location="backend/tasks/post_import_pipeline.py:226">
P2: `categorization_method` is incorrectly hardcoded to `llm` for results that can come from user overrides.</violation>
</file>

<file name="backend/app/services/category_embedding.py">

<violation number="1" location="backend/app/services/category_embedding.py:115">
P1: Embedding output dimensions are not pinned to the database vector size, so changing the embedding model can generate incompatible vectors and break categorization writes/queries.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

[cubic-dev-ai]

3 issues found across 4 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="backend/scripts/backfill_transaction_embeddings.py">

<violation number="1" location="backend/scripts/backfill_transaction_embeddings.py:49">
P2: `q.all()` loads every matching transaction into memory, which can cause high memory usage for large backfills despite the batching loop.</violation>

<violation number="2" location="backend/scripts/backfill_transaction_embeddings.py:95">
P2: Validate `--batch-size` as a positive integer; otherwise invalid values can crash the script at runtime.</violation>
</file>

<file name="backend/app/services/category_embedding.py">

<violation number="1" location="backend/app/services/category_embedding.py:121">
P2: `dimensions` is sent unconditionally, but OpenAI only supports that parameter for `text-embedding-3*` models. If `CATEGORIZATION_EMBEDDING_MODEL` is set to `text-embedding-ada-002`, embedding requests fail and the service silently falls back by returning empty vectors.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

[gianniskotsas]

@cubic-dev-ai these five issues have already been addressed in earlier commits on this branch:

1. post_import_pipeline.py — categorization_method hardcoded to llm → fixed in 52dc736. The function is now structured as three explicit tiers (override / embedding / llm) and tags categorization_method correctly per tier (see lines 238, 246, 254 of the current file).
2. category_embedding.py — dimensions not pinned → fixed in 52dc736 by passing dimensions=EMBEDDING_DIMENSIONS and validating returned vector length post-call.
3. category_embedding.py — dimensions sent unconditionally → refined in ef529b3: only passed when EMBEDDING_MODEL.startswith('text-embedding-3'), so text-embedding-ada-002 etc. won't 400.
4. backfill_transaction_embeddings.py — q.all() memory → fixed in ef529b3 by selecting only IDs upfront and batch-loading rows per chunk with expunge_all() between iterations.
5. backfill_transaction_embeddings.py — --batch-size validation → fixed in ef529b3 via a custom _positive_int argparse type applied to both --batch-size and --limit.

Looks like the review re-scanned the original commits in the diff. No further action needed.

[cubic-dev-ai]

@cubic-dev-ai these five issues have already been addressed in earlier commits on this branch:

1. post_import_pipeline.py — categorization_method hardcoded to llm → fixed in 52dc736. The function is now structured as three explicit tiers (override / embedding / llm) and tags categorization_method correctly per tier (see lines 238, 246, 254 of the current file).
2. category_embedding.py — dimensions not pinned → fixed in 52dc736 by passing dimensions=EMBEDDING_DIMENSIONS and validating returned vector length post-call.
3. category_embedding.py — dimensions sent unconditionally → refined in ef529b3: only passed when EMBEDDING_MODEL.startswith('text-embedding-3'), so text-embedding-ada-002 etc. won't 400.
   ...

@gianniskotsas You're right — looking at the current code state in the latest commits, all five of those issues have been addressed exactly as described.

It appears my previous feedback was based on an earlier iteration of the changes before those fixes were pushed.

• Categorization Tiers: post_import_pipeline.py correctly tags the method as override, embedding, or llm based on which tier resolved the transaction.
• Embedding Safety: CategoryEmbeddingService now correctly pins dimensions only for text-embedding-3 models and includes the post-call length check to prevent DB mismatches.
• Memory Efficiency: The backfill script's ID-first approach and use of expunge_all() properly protects against OOM issues on large datasets.
• Input Validation: The _positive_int argparse type ensures valid runtime flags.

The implementation looks solid and follows the requested best practices. No further action is needed.