ai-issue-triage

Semantic issue triage and RAG-based impact analysis pipeline that runs fully local — no LLM key required.

ai-issue-triage ingests issues from an issue tracker (GitHub Issues or bundled sample data), classifies them with pgvector semantic search, and injects similar past cases as RAG context so an LLM can produce a structured impact analysis (risk_score, findings, summary, citations). Without an LLM key it degrades gracefully to a rule-based analyzer, so the whole pipeline runs with a single docker compose up.

Why this exists

Keyword rules don't understand "the login screen hangs after SSO" and "auth redirect never returns" are the same problem. ai-issue-triage uses embeddings so classification and analysis are driven by meaning, and it grounds the LLM in previously resolved cases instead of letting it guess.

Highlights

• Search drives quality — semantic KNN retrieves past resolved cases that are injected into the analysis prompt (RAG), not brittle keyword matching.
• Runs with no API key — local sentence-transformers embeddings + a rule-based analyzer fallback. LLM (Claude/OpenAI) is opt-in.
• Production-shaped, not a toy — idempotent incremental ingestion, classification confidence + source tracking, structured analysis output with citations, and SQL migrations.
• Function-level code search — tree-sitter parses Python/Java into symbol-level chunks (functions, methods, classes), embedded for semantic "which function relates to this?" search (/code/search).

Architecture

flowchart TD
    SRC["GitHub Issues / sample.jsonl"] -->|"ingest (idempotent upsert)"| DB[("PostgreSQL + pgvector")]
    DB -->|"embed locally (sentence-transformers, 384d)"| DB
    DB -->|"self-excluded KNN vote + keyword rules"| CLS["Classification<br/>(category, confidence, source)"]
    CLS -->|"retrieve resolved cases"| RAG["RAG context<br/>(top-k similar cases)"]
    RAG --> Q{"LLM key set?"}
    Q -->|yes| AN["Analysis<br/>(risk_score, findings, citations)"]
    Q -->|"no"| FB["Rule-based fallback"] --> AN
    AN --> API["FastAPI REST + HTMX dashboard"]

Loading

Key design decisions (full rationale in docs/design-decisions.md):

• pgvector over a dedicated vector DB — one datastore for vectors and relational data; trivial joins, nothing extra to operate.
• Local embeddings + rule fallback — the whole pipeline runs with no API key or cost; an LLM is opt-in and degrades gracefully if it fails.
• Hybrid classification (KNN + keyword) — semantic vote over a labeled case library, backed by keyword rules for cold-start phrasings; the deciding source is stored for explainability.
• LLM behind a Protocol — Claude / OpenAI / offline analyzers are interchangeable; the pipeline shape never changes.

More: docs/architecture.md · docs/rag-strategy.md.

Quick start

cp .env.example .env      # works as-is; leave LLM_PROVIDER=none for offline mode
make up                   # postgres(pgvector) + api  (first run downloads the model)
make seed                 # load 200 sample issues + 50 resolved cases, then embed
make pipeline             # ingest -> embed -> classify -> RAG analyze
# open http://localhost:8000/   (dashboard)
curl 'localhost:8000/search?q=login%20timeout&k=5' | python -m json.tool
curl  localhost:8000/issues/1/analysis | python -m json.tool

# (optional) function-level semantic code search over this repo's own source
make index-code
curl 'localhost:8000/code/search?q=validate%20a%20session%20token&k=3' | python -m json.tool

# hybrid search (vector + full-text via RRF) and pipeline stats
curl 'localhost:8000/search?q=double%20charge&mode=hybrid&k=5' | python -m json.tool
curl  localhost:8000/stats | python -m json.tool

Enable an LLM (optional):

# in .env
LLM_PROVIDER=anthropic
ANTHROPIC_API_KEY=sk-...
# docker compose restart api

Continuous integration

Every push runs the full pipeline in GitHub Actions against a real Postgres + pgvector service container (.github/workflows/ci.yml): it seeds the sample data, runs classify + RAG-analyze end to end, evaluates the classifier against ground-truth labels (make eval → accuracy / macro-F1 / confusion matrix), indexes code, and executes the test suite — all with the offline rule-based analyzer (no API key). A separate lint job enforces ruff (lint + format) and mypy (types); a .pre-commit-config.yaml runs the same checks locally. Unit tests also run locally without Docker:

pip install -e ".[dev]"
pytest -q tests/test_classifier.py tests/test_rule_fallback.py

Out of scope

By design this is a focused demo, not a platform. The following are intentionally not included: source-repo discovery / diff / automated code fixes, CI / build pipelines, authentication / multi-tenancy, a real-time scheduler (incremental ingest exists but is triggered manually), a SPA frontend (a single HTMX page is used instead), and attachment OCR.

License

MIT — see LICENSE.